Video image processing method and device, electronic device, and storage medium

Abstract

The present application discloses a method and device for processing video images, an electronic device, and a storage medium, wherein the method includes: obtaining N frames of first image frames; using a first network module to perform super-resolution reconstruction on each of the N frames of first image frames to obtain N frames of second image frames; using a second network module to perform alignment and fusion processing on the N frames of second image frames to obtain result frames.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for processing video images, an electronic device, and a storage medium. Background Art

[0002] Regarding the processing of video images, in some schemes, multiple image frames in the video are simply sent directly into a network module for training. When this scheme trains multiple image frames, the blur, noise, downsampling and compression in the multiple image frames will increase the difficulty of network module training and cannot make the final video image achieve a good display effect. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present application provide a method and device for processing video images, an electronic device, and a storage medium.

[0004] The embodiment of the present application provides a method for processing a video image, the method comprising:

[0005] Get the first image frame of N frames;

[0006] Using the first network module, super-resolution reconstruction is performed on each of the N first image frames to obtain N second image frames;

[0007] The second network module is used to perform alignment and fusion processing on the N frames of second image frames to obtain a result frame.

[0008] The embodiment of the present application also provides a video image processing device, the device comprising:

[0009] An input module, used for acquiring N first image frames, and inputting the N first image frames into a first module;

[0010] A first network module is used to perform super-resolution reconstruction on each of the N first image frames to obtain N second image frames;

[0011] The second network module is used to perform alignment and fusion processing on the N frames of second image frames to obtain a result frame.

[0012] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the video image processing method described in the above embodiment.

[0013] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the video image processing method described in the above embodiment.

[0014] The technical solution of the embodiment of the present application is to obtain N frames of first image frames; use the first network module to perform super-resolution reconstruction on each frame of the N frames of first image frames to obtain N frames of second image frames; use the second network module to perform alignment and fusion processing on the N frames of second image frames to obtain result frames. In this way, the first network module can be used to perform super-resolution reconstruction on the acquired low-quality N frames of first image frames, reduce the noise and blur of the N frames of first image frames, and improve the quality of each frame of the N frames of first image frames; after the first network module outputs high-quality N frames of second image frames, when the second network module is used to process the N frames of second image frames, the interference of the noise contained in the initial N frames of first image frames can be isolated, so that the second module can focus on realizing the fusion and utilization of effective information in multiple frames of second image frames, and improve the quality of the image frames finally obtained. In addition, by dividing the image processing process of the N frames of first image frames into two sub-networks of the first network module and the second network module, the flexibility of the entire image processing task can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Several video image processing methods provided in the embodiments of the present application;

[0016] Figure 2 A schematic diagram of the display effect of three frames of images taken by a terminal device;

[0017] Figure 3 A schematic diagram of a video image processing method provided in an embodiment of the present application;

[0018] Figure 4 The video image processing process provided in the embodiment of the present application is shown in FIG. Figure 1 ;

[0019] Figure 5 The video image processing process provided in the embodiment of the present application is shown in FIG. Figure 2 ;

[0020] Figure 6 Schematic diagram of the first network module provided in the embodiment of the present application Figure 1 ;

[0021] Figure 7 Schematic diagram of the first network module provided in the embodiment of the present application Figure 2 ;

[0022] Figure 8Schematic diagram of the second network module provided in the embodiment of the present application Figure 1 ;

[0023] Fig. 9 Schematic diagram of the second network module provided in the embodiment of the present application Figure 2 ;

[0024] Fig.10 A schematic diagram of the structural composition of a video image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable a more detailed understanding of the features and technical contents of the present application, the implementation of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the present application.

[0026] Figure 1 Several video image processing methods are listed in the embodiments of this application, such as Figure 1 As shown in the figure, video image processing algorithms are mainly divided into two categories, one is the video image processing algorithm based on convolutional neural network (CNN), and the other is the traditional video image processing algorithm without CNN. At present, there are usually two ideas when studying video image processing technology. One is to reconstruct each image frame in the video as a separate image, and then splice each reconstructed frame together to form the final result video; the other is to use the time domain characteristics of the video. When reconstructing a certain image frame, multiple image frames before and after the image frame in the video are used to develop the algorithm.

[0027] For the processing of video images, in a specific scheme, the processing of video images is achieved by using a convolutional neural network with two-stage motion compensation. The scheme specifically includes the following steps: 1. Accepting a video with a first plurality of frames, the first plurality of frames having a first resolution; 2. Generating a plurality of deformed frames from the first plurality of frames based on a first type of motion compensation; 3. Generating a second plurality of frames with a second resolution, wherein the second resolution is a higher resolution than the first resolution; 4. Using a convolutional neural network to obtain each frame of the second plurality of frames with a second resolution from a subgroup of the plurality of deformed frames; 5. Generating a third plurality of frames with a second resolution based on a second type of motion compensation; 6. Obtaining each frame of the third plurality of frames with a second resolution from the fusion of the subgroups of the second plurality of frames. This scheme requires motion compensation for both low-resolution image frames and high-resolution image frames, which increases the complexity of the algorithm and results in poor convergence performance of the network. Moreover, this scheme is not suitable for scenarios where 1:1 video quality enhancement is required.

[0028] In another specific solution, the reconstruction of video images is mainly based on residual learning and implicit motion compensation. The solution specifically includes the following steps: 1. Train convolutional neural network models with different magnifications respectively; 2. Take adjacent low-resolution image frames as input, and obtain the final reconstruction result through the network model trained in the previous step. This solution simply feeds the three adjacent frames in the video into the CNN network, and cannot fully utilize the multi-frame information in the video.

[0029] In the above two solutions, when using multiple image frames in a video to obtain a reconstructed image frame, the multiple image frames obtained from the video are simply directly input into a network module, and the network module is used to train the multiple image frames at the same time. Figure 2 As shown, there are often multiple types of degradation in the image frames captured and generated by the terminal device, such as blur, noise, downsampling and compression, etc. There is obvious blur and noise in the image frames when displayed. If a network module is used to directly fuse multiple image frames including noise and blur and perform subsequent feature learning and training, the blur and high-intensity noise in the image frames will increase the difficulty of the subsequent network processing process, causing the network module to fall into a local optimal solution, and even making the training process of the network module unable to converge, reducing the quality of the image frames finally trained using the network module.

[0030] Below, based on the problems existing in the above solutions, technical solutions of various embodiments of the present application are proposed.

[0031] Figure 3 A schematic diagram of a process flow of a video image processing method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the method comprises the following steps:

[0032] Step 301: Acquire N first image frames.

[0033] In the embodiment of the present application, N first image frames are obtained from the video to be processed, and the value of N can be extended to any integer greater than or equal to 1. The N first image frames can be N consecutive first image frames in the video to be processed, or can be N discontinuous first image frames selected from M consecutive first image frames (M is an integer greater than N). Here, if the N first image frames are discontinuous, it is necessary to ensure that the time interval between any two first image frames in the N first image frames is less than the set time threshold.

[0034] In an embodiment of the present application, the N first image frames can be directly obtained from the video to be processed, and can be the original N image frames in the video to be processed, or can be N image frames obtained after the original N image frames are processed in a certain way.

[0035] In a specific implementation, N first image frames may be acquired in the following manner: acquiring a video to be processed, and acquiring N third image frames from the video; extracting image data of a specific channel from the N third image frames to obtain the N first image frames.

[0036] Specifically, after obtaining the original N frames of third image frames from the video to be processed, image data of a specific channel can be extracted from the N frames of third image frames to obtain N frames of first image frames containing only image data of the specific channel. When the first image frames are subsequently processed using the first network module, the feature information contained in the N frames of first image frames containing only the specific channel can be utilized.

[0037] Here, in a preferred embodiment, for an image frame whose color space is YUV, the specific channel may be the Y channel. When processing the image frame, the feature data of the Y channel has a greater impact on the image processing result, while the U channel and the V channel have a smaller impact on the image processing result. Therefore, only the image data of the Y channel may be input into the first network module, and the final result frame may be obtained after being processed by the subsequent second network module. In this way, the amount of data processing in the image processing process may be reduced.

[0038] It should be noted that the embodiment of the present application does not limit the number of channels of the image frame input to the first network module. For example, the embodiment of the present application can also directly input the image frame containing multiple channels (such as RGB three channels) into the first network module, and obtain the final result frame through subsequent processing by the second module.

[0039] In one implementation, after acquiring N first image frames, it is also necessary to select a first key frame from the N first image frames; the first key frame is an image frame in the N first image frames.

[0040] Here, the selection of the first key frame can be determined according to the specific application scenario, and the first key frame can be any one of the N first image frames. If the N first image frames are N image frames obtained from the video, then the N first image frames are in chronological order. By sorting the N first image frames from 1 to N in chronological order, the first key frame can be selected from the N first image frames according to the order. Exemplarily, when the value of N is 5, the middle image frame of the five first image frames, that is, the third image frame, can be selected as the first key frame, or the second image frame or the fourth image frame of the five first image frames can be selected as the first key frame. It should be noted that, generally, for a specific application scenario, the value of N and the order of the first key frame relative to the first image frame of N frames are generally unchanged. For example, for the same video A to be processed, if the value of N is set to 5, and the selected first key frame is the second frame of the five frames sorted in time, then when performing image processing on video A, the value of N is 5, and the second frame of the five frames is selected as the first key frame.

[0041] Step 302: Use the first network module to perform super-resolution reconstruction on each of the N first image frames to obtain N second image frames.

[0042] Figure 4 The video image processing flow when the value of N is 3 provided in the embodiment of the present application; Figure 5 The video image processing flow when the value of N is 5 provided in the embodiment of the present application; Figure 4 and Figure 5 In the example, network A represents the first network module and network B represents the second network module.

[0043] In the embodiment of the present application, the first network module is mainly used to perform single-frame super-resolution reconstruction on each of the N first image frames, and restore the received low-resolution first image frame to a second image frame with higher definition. The first network module may select a classic single image super-resolution reconstruction (SISR, Single Image Super Resolution) method.

[0044] In an optional implementation manner of the present application, the first network module includes a residual network. When the first network module includes the residual network, the first network module is used to perform super-resolution reconstruction on each of the N first image frames to obtain the N second image frames. The specific process is:

[0045] For each first image frame in the N frames of first image frames, feature learning is performed on the first image frame through the residual network to obtain a feature image corresponding to the first image frame; the first image frame is spliced ​​with the feature image to obtain a second image frame.

[0046] Specifically, Figure 6 As shown, the first network module may specifically include a residual network. Figure 6 In the figure, a low-resolution (LR) video frame represents a first image frame among N first image frames, and an initial super-resolution reconstruction (SR) frame represents an image frame obtained by super-resolution reconstruction of a single frame of the first image frame. By using a residual network to super-resolution reconstruct the LR video frame, an initial SR frame after super-resolution reconstruction of the LR video frame can be obtained.

[0047] Here, the splicing of the first image frame and the feature image is achieved through a global residual structure. Introducing the global residual structure in the first network module can reduce the difficulty of training the first network module.

[0048] In another optional embodiment of the present application, the first network module includes a multi-level feature extraction network; when the first network module includes a multi-level feature extraction network, the specific process of using the first network module to perform super-resolution reconstruction on each of the N first image frames to obtain N second image frames is:

[0049] For each first image frame in the N first image frames, feature learning is performed on the first image frame respectively through the multi-level feature extraction network to obtain a plurality of feature images with different scales; and the plurality of feature images with different scales are spliced ​​to obtain a second image frame.

[0050] Figure 7 The first network module includes a schematic diagram of a multi-level feature extraction network, which is also called a pyramid network. After the multi-level feature extraction network is introduced into the first network module, multi-scale information of the LR video frame can be obtained by downsampling. The embodiment of the present application does not limit the number of downsampling. In different application scenarios, the number of downsampling can be different. Taking two downsampling as an example, Figure 7 In the method, after two downsamplings, feature learning is performed at different scales. At the output end of the multi-level feature extraction network, low-scale information is sampled in the form of deconvolution and then fused with high-scale features. This can ultimately achieve effective utilization of multi-scale information in LR video frames. This neural network structure that combines multi-scale information can reduce the time complexity of the first network module.

[0051] In an optional implementation manner of the present application, the first network module includes a first amplification module, and the first amplification module is used to improve the resolution of the image frame input to the first amplification module.

[0052] Specifically, Figure 6 As shown, the first network module may also include an upsampling layer. In the process of training the LR video frame using the first network module, by setting the upsampling layer, the resolution of the image frame input to the upsampling layer can be increased according to a specified ratio, so that the resolution of the initial SR frame output by the first network module is higher than the resolution of the LR video frame input to the first network module.

[0053] Step 303: Use the second network module to perform alignment and fusion processing on the N frames of second image frames to obtain a result frame.

[0054] Specifically, after using the first network module to perform single-frame super-resolution reconstruction on each of the N first image frames to obtain N second image frames, the second network module can be used to align and fuse the N second image frames to obtain the final result frame. Here, when the value of N is greater than 2, when using the second network module to align and fuse the N second image frames, the N second image frames can be input into the second network module at the same time, and the second network module can simultaneously align and fuse the N second image frames to obtain the reconstructed result frame. The N second image frames can also be grouped in pairs, and the second network module can align and fuse every two second image frames.

[0055] In a preferred embodiment, the specific process of using the second network module to perform alignment and fusion processing on the N frames of second image frames to obtain the result frame is:

[0056] Using the second network module, each second image frame in N-1 frames of second image frames is aligned and fused with the second key frame to obtain N-1 first fused frames; the N-1 frames of second image frames are image frames in the N frames of second image frames except the second key frame;

[0057] Using the second network module, aligning and fusing every two first fused frames in the N-1 first fused frames to obtain M second fused frames, where M is an integer greater than or equal to 1 and less than N-1;

[0058] A result frame is obtained based on the M second fused frames.

[0059] Here, the process of obtaining the result frame based on the M second fused frames can be implemented in the following manner:

[0060] When M is equal to 1, the second fused frame is determined as the result frame;

[0061] When M is greater than 1, the second network module is used to perform alignment and fusion processing on every two second fused frames in the M second fused frames; if the number of obtained fused frames is 1, the obtained fused frame is determined as the result frame; if the number of obtained fused frames is greater than 1, the second network module is continued to be used to perform alignment and fusion processing on every two fused frames in the multiple fused frames obtained until the number of obtained fused frames is 1.

[0062] It should be noted that when the second network module is used to align and fuse image frames, if three or more image frames are directly input to the second network module so that the second network module network can achieve alignment and fusion of three or more image frames, the training difficulty of the second network module will increase, making it difficult for the second network module to converge and easy to fall into a local optimal solution. The embodiment of the present application can reduce the training difficulty of the second network module and improve the effect of aligning and fusing two image frames by allowing the second network module to receive only two image frames as input.

[0063] Figure 4 When the value of N is 3, after sorting the three first image frames, the middle image frame is selected as the first key frame (that is, the value of i is 2), and the three first image frames are processed by network A and network B to obtain the processing process of the result frame.

[0064] Figure 5 This is the process of sorting the 5 first image frames when the value of N is 5, selecting the middle image frame as the first key frame (that is, the value of i is 3), and using network A and network B to process the 5 first image frames to obtain the result frame.

[0065] It should be noted that when the second network module is used to further fuse the fused frames generated by the second network module in the previous stage, if there are remaining single-frame fused frames that have not been grouped after the fused frames generated in the previous stage are combined in pairs, the fused frame can be discarded, or any other fused frame can be selected to be combined with the fused frame, and the second network can be used to further align and fuse the two fused frames.

[0066] In the embodiment of the present application, the second network module can be implemented by means of a convolutional neural network, or by traditional algorithms such as traditional block matching and homography matrix.

[0067] In an optional implementation manner of the present application, the second network module includes: a splicing module and a fusion module;

[0068] The splicing module is used to splice two image frames input to the splicing module in the target dimension, and output feature data of the two spliced ​​image frames;

[0069] The fusion module is used to fuse the spliced ​​feature data and output a fused frame corresponding to the two image frames.

[0070] Here, in addition to the splicing module and the fusion module, the second network module also includes an alignment module, which is located between the splicing module and the fusion module, or before the splicing module and the fusion module. The alignment module is used to align two image frames input into the alignment module and output feature data of the two aligned image frames.

[0071] When the alignment module is located between the splicing module and the fusion module, the specific processing process of the second network module for the two image frames is: using the splicing module to splice the two image frames input to the splicing module on the target dimension, and outputting the feature data of the two spliced ​​image frames; using the alignment module to align the spliced ​​feature data, and outputting the aligned feature data of the two image frames; using the fusion module to fuse the aligned feature data, and outputting a fused frame corresponding to the two image frames.

[0072] Specifically, Figure 8 As shown, the second network module outputs a fused frame corresponding to the two received image frames in the processing order of "splicing-aligning-fusion".

[0073] for Figure 8 In the second network module shown, the stitching module first stitches the two input image frames in the third dimension, and then sends the stitched data to the alignment module and the fusion module for learning in sequence. Since each convolutional layer in the alignment module and the fusion module has several filters, the two image frames can be aligned and fused through several filters in the two network modules, and the fused frame corresponding to the two image frames is output.

[0074] It should be noted that the two image frames here represent different image frames at different stages. Exemplarily, when the second network module is used for the first time to perform alignment and fusion processing on the two image frames, the two image frames refer to the second key frame corresponding to the first key frame and one second image frame among the other N-1 second image frames. When the second network module is used for alignment and fusion processing for the second time and thereafter, the two image frames refer to the two fused frames included in the combination after grouping in the multiple fused frames generated in the previous stage.

[0075] In an embodiment of the present application, the target dimension refers to the image channel dimension other than the width and height in the image frame. For example, for a color image with RGB three channels, the dimension where the three color channels R (i.e. red), G (i.e. green) and B (i.e. blue) are located is the third dimension. At this time, the stitching process of the stitching module can be understood as inputting three images with three channels of R, G, and B respectively. After the stitching module performs stitching in the third dimension, a normal RGB three-channel color image can be output.

[0076] Here, by utilizing a second network module composed of a stitching module, an alignment module, and a fusion module cascaded in sequence, the information of two image frames input into the second network module can be effectively fused and utilized in the processing order of "stinging-alignment-fusion".

[0077] When the alignment module is located before the splicing module and the fusion module, the specific processing process of the second network module for the two image frames is: use the alignment module to align the two image frames input to the alignment module, and output the feature data of the two aligned image frames; use the splicing module to splice the aligned feature data on the target dimension, and output the feature data of the two spliced ​​image frames; use the fusion module to fuse the spliced ​​feature data, and output a fused frame corresponding to the two image frames.

[0078] Specifically, Fig. 9 As shown, the second network module processes the two image frames in the order of "alignment-splicing-fusion" and outputs a fused frame corresponding to the two image frames.

[0079] for Fig. 9 As shown in the alignment and fusion module, the alignment module first aligns the two input image frames through several layers of convolution structures. The two aligned image frames are then spliced ​​in the third dimension by the splicing module, and then input into the convolution structure corresponding to the subsequent fusion module to complete the fusion of the two image frames, and output a fused frame corresponding to the two image frames.

[0080] Here, by utilizing a second network module composed of an alignment module, a stitching module, and a fusion module cascaded in sequence, the feature information of two image frames input into the second network module can be effectively fused and utilized in the processing order of "alignment-spinning-fusion".

[0081] In an embodiment of the present application, in order to achieve better alignment, in the process of aligning each frame, some special convolution structures, such as a hole convolution layer, a deformable convolution layer, etc., can be used in the alignment module to improve the effect of aligning two image frames.

[0082] In an optional implementation manner of the present application, the second network module further includes a second amplification module, and the second amplification module is used to increase the resolution of the image frame input to the second amplification module.

[0083] Specifically, in an embodiment of the present application, in addition to setting a first amplification module in the first network module to improve the resolution of the image frame, a second amplification module can also be set in the second network module according to the specific application scenario to improve the resolution of the image frame input into the second amplification module.

[0084] It should be noted that, in the embodiment of the present application, the resolution increase factor of the result frame finally generated relative to the N second image frames input to the second network module is associated with the value of N and the specific magnification set by the amplification module. Figure 4 As shown in the figure, when the value of N is 3, two rounds of alignment and fusion of the second network module are required to obtain the final result frame. At this time, if the magnification of the second magnification module is 2, the improvement ratio of the final result frame relative to the N-frame second image frame is 4; Figure 5 In the case where the value of N is 5, three rounds of alignment and fusion of the second network module are required to obtain the final result frame. At this time, if the magnification of the second magnification module is 2, the improvement ratio of the final result frame relative to the N-frame second image frame is 8.

[0085] Here, it should be noted that the first amplification module in the first network module and the second amplification module in the second network module in the embodiment of the present application can be flexibly set according to the target magnification. If the entire video image processing process does not require the image frame to be magnified, the first amplification module set in the first network module can be cancelled, and the second amplification module set in the second network module can be cancelled. At this time, it is equivalent to using the first network module and the second network module to reconstruct the first key frame in proportion. The first amplification module and the second amplification module are respectively set in the first network module and the second network module according to the needs, which can improve the clarity of the first key frame while improving the resolution of the first key frame, and can be applied to various scenes that require super-resolution reconstruction of image frames in the video.

[0086] In an optional implementation manner of the present application, the training of the first network module and the second network module may be carried out in the following manner:

[0087] Using a first training data set to train the first network module to obtain a trained first network module;

[0088] The trained first network module is used to determine a second training data set for training the second network module, and the second training data set is used to train the second network module to obtain a trained second network module.

[0089] Specifically, when training the first network module and the second network module, the first network module can be trained using the SISR data set. After the first network module converges, a new version of the training data set for training the second network module is generated in combination with the trained first network module, thereby ensuring that when training the second network module, the input of the second network module is a clear image frame.

[0090] In an embodiment of the present application, the first training data set is a data set used to train the first network module, and the second training data set is a data set used to train the second network module; wherein the second training data set may be a data set obtained by inputting the first training data set into the trained first network module, or may be a data set obtained by processing the third data set using the trained first network module; wherein the third data set and the first training data set are different data sets.

[0091] Here, taking the deep learning framework of Pytorch as an example, the input during training and testing are both tensors with four dimensions, that is, quantity × channel × width × height. When the solution of the present invention is used, in order to reduce the time consumption of network deployment, the first network module directly receives an image frame with a data volume of 3×1×640×360 (taking 3 frames of input, only processing the Y channel of the 360p image frame as an example), and the data volume of the image frame of the final output result is 3×1×1920×1080 (3 times magnification). Afterwards, the input received by the second network module is similar to that of the first network module, and the specific input specification depends on the network structure and the actual application scenario.

[0092] Here, by dividing the reconstruction task of the image frame in the video into two sub-network modules (i.e., the first network module and the second network module), each network module completes the corresponding function. During training, by training the two network modules separately, both network modules can obtain excellent convergence performance, and finally the two trained network modules are cascaded to obtain a better reconstruction effect of the video image frame. In addition, by using the high-quality image frames output by the first network module, the training difficulty of the second network module can be reduced, and the reconstruction effect of the image frames in the video can be improved.

[0093] The technical solution of the embodiment of the present application can make full use of the time domain information of multiple image frames in the video, divide the reconstruction task of low-quality image frames in the video into two sub-network modules, namely the first network module and the second network module, and use the first network module to perform SISR processing on each image frame in the low-quality multiple image frames, reduce the noise and blur in the high-quality multiple image frames, and improve the clarity of each image frame in the low-quality multiple image frames, and then send the processed image frames to the second network module, which aligns and fuses the processed image frames. The second module can isolate the interference of noise and other problems in the image frames during the image alignment and fusion process, and focus on realizing the effective fusion and utilization of multi-frame information; in addition, by dividing the processing process of the image frames in the video into two sub-networks, namely the first network module and the second network module, the flexibility of the entire video image processing task can be improved.

[0094] Fig.10 A schematic diagram of the structure of a video image processing device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the video image processing device includes:

[0095] An input module 1001 is used to obtain N first image frames and input the N first image frames into a first module;

[0096] The first network module 1002 is used to perform super-resolution reconstruction on each of the N first image frames to obtain N second image frames;

[0097] The second network module 1003 is used to perform alignment and fusion processing on the N frames of second image frames to obtain a result frame.

[0098] In an optional implementation manner of the present application, the device further includes:

[0099] A selection module 1004 is used to select a first key frame from the N first image frames, where the first key frame is an image frame in the N first image frames;

[0100] The first key frame is reconstructed through super-resolution by the first network module to obtain a second key frame, and the second key frame is an image frame among the N frames of second image frames.

[0101] In an optional implementation of the present application, the second network module 1003 is specifically used to: use the second network module to align and fuse each second image frame in the N-1 frames of second image frames with the second key frame to obtain N-1 first fused frames; the N-1 frames of second image frames are image frames other than the second key frame in the N frames of second image frames; use the second network module to align and fuse every two first fused frames in the N-1 first fused frames to obtain M second fused frames; M is an integer greater than or equal to 1 and less than N-1; and obtain a result frame based on the M second fused frames.

[0102] In an optional implementation of the present application, the second network module 1003 is also specifically used to: when M is equal to 1, determine the second fused frame as a result frame; when M is greater than 1, use the second network module to perform alignment and fusion processing on every two second fused frames in the M second fused frames; if the number of obtained fused frames is 1, determine the obtained fused frame as a result frame; if the number of obtained fused frames is greater than 1, continue to use the second network module to perform alignment and fusion processing on every two fused frames in the multiple fused frames obtained until the number of obtained fused frames is 1.

[0103] In an optional implementation manner of the present application, the first network module 1002 includes a residual network;

[0104] The first network module 1002 is specifically used for: for each first image frame in the N frames of first image frames, performing feature learning on the first image frame through the residual network to obtain a feature image corresponding to the first image frame; and splicing the first image frame with the feature image to obtain a second image frame.

[0105] In an optional implementation manner of the present application, the first network module 1002 includes a multi-stage feature extraction network;

[0106] The first network module 1002 is specifically used for: for each first image frame in the N frames of first image frames, performing feature learning on the first image frame respectively through the multi-level feature extraction network to obtain multiple feature images with different scales; and splicing the multiple feature images with different scales to obtain a second image frame.

[0107] In an optional implementation manner of the present application, the first network module 1002 includes a first amplification module, and the first amplification module is used to increase the resolution of the image frame input to the first amplification module.

[0108] In an optional implementation manner of the present application, the second network module 1003 includes: a splicing module and a fusion module;

[0109] The splicing module is used to splice the two image frames input to the splicing module in the target dimension and output the feature data of the two image frames after splicing;

[0110] The fusion module is used to fuse the spliced ​​feature data and output a fused frame corresponding to the two image frames.

[0111] In an optional implementation manner of the present application, the second network module 1003 further includes: an alignment module; the alignment module is located between the splicing module and the fusion module, or the alignment module is located before the splicing module;

[0112] The alignment module is used to align two image frames input to the alignment module, and output feature data of the two aligned image frames.

[0113] In an optional implementation manner of the present application, the device further includes:

[0114] A first training module 1005 is used to train the first network module using a first training data set to obtain a trained first network module;

[0115] The second training module 1006 is used to determine a second training data set for training the second network module using the trained first network module, and train the second network module using the second training data set to obtain a trained second network module.

[0116] In an optional implementation of the present application, the input module 1001 is specifically used to: obtain a video to be processed, and obtain N frames of third image frames from the video; extract image data of a specific channel from the N frames of third image frames to obtain the N frames of first image frames.

[0117] Those skilled in the art should understand that Fig.10 The implementation functions of each module in the video image processing device shown can be understood by referring to the relevant description of the aforementioned video image processing method. Fig.10 The functions of the modules in the video image processing device shown can be implemented by a program running on a processor, or by a specific logic circuit.

[0118] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the video image processing method described in the above embodiment.

[0119] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the video image processing method described in the above embodiment is implemented.

[0120] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0121] In the several embodiments provided in the present application, it should be understood that the disclosed methods and intelligent devices can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0122] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0123] In addition, all functional units in the embodiments of the present application may be integrated into a second processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0124] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for processing a video image, characterized in that: The method comprises: Get the first image frame of N frames; Using the first network module, super-resolution reconstruction is performed on each of the N first image frames to obtain N second image frames; Using the second network module to perform alignment and fusion processing on the N frames of second image frames to obtain a result frame; The method further comprises: Selecting a first key frame from the N first image frames, where the first key frame is an image frame among the N first image frames; The first key frame is reconstructed by super-resolution through the first network module to obtain a second key frame, and the second key frame is an image frame among the N frames of second image frames; The step of using the second network module to perform alignment and fusion processing on the N frames of second image frames to obtain a result frame includes: Using the second network module, each second image frame in N-1 frames of second image frames is aligned and fused with the second key frame to obtain N-1 first fused frames; the N-1 frames of second image frames are image frames in the N frames of second image frames except the second key frame; Using the second network module, aligning and fusing every two first fused frames in the N-1 first fused frames to obtain M second fused frames, where M is an integer greater than or equal to 1 and less than N-1; A result frame is obtained based on the M second fused frames.

2. The method according to claim 1, characterized in that The obtaining a result frame based on the M second fused frames includes: When M is equal to 1, the second fused frame is determined as the result frame; When M is greater than 1, the second network module is used to perform alignment and fusion processing on every two second fused frames in the M second fused frames; if the number of obtained fused frames is 1, the obtained fused frame is determined as the result frame; if the number of obtained fused frames is greater than 1, the second network module is continued to be used to perform alignment and fusion processing on every two fused frames in the multiple fused frames obtained until the number of obtained fused frames is 1.

3. The method according to claim 1, characterized in that The first network module includes a residual network; wherein, The method of using the first network module to perform super-resolution reconstruction on each of the N first image frames to obtain N second image frames includes: For each first image frame in the N first image frames, performing feature learning on the first image frame through the residual network to obtain a feature image corresponding to the first image frame; The first image frame is spliced ​​with the characteristic image to obtain a second image frame.

4. The method according to claim 1, characterized in that: The first network module includes a multi-level feature extraction network; wherein, The method of using the first network module to perform super-resolution reconstruction on each of the N first image frames to obtain N second image frames includes: For each first image frame in the N first image frames, performing feature learning on the first image frame respectively through the multi-level feature extraction network to obtain a plurality of feature images with different scales; The plurality of feature images with different scales are spliced ​​to obtain a second image frame.

5. The method according to any one of claims 1 to 4, characterized in that The first network module includes a first amplification module, and the first amplification module is used to improve the resolution of the image frame input to the first amplification module.

6. The method according to any one of claims 1 to 4, characterized in that The second network module includes: a splicing module and a fusion module; The splicing module is used to splice two image frames input to the splicing module in the target dimension, and output feature data of the two spliced ​​image frames; The fusion module is used to fuse the spliced ​​feature data and output a fused frame corresponding to the two image frames.

7. The method according to claim 6, characterized in that The second network module further includes: an alignment module; the alignment module is located between the splicing module and the fusion module, or the alignment module is located before the splicing module; The alignment module is used to align two image frames input to the alignment module, and output feature data of the two aligned image frames.

8. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Using a first training data set to train the first network module to obtain a trained first network module; The trained first network module is used to determine a second training data set for training the second network module, and the second training data set is used to train the second network module to obtain a trained second network module.

9. The method according to any one of claims 1 to 4, characterized in that The acquiring of N first image frames comprises: Acquire a video to be processed, and acquire N third image frames from the video; Image data of a specific channel is extracted from the N frames of third image frames to obtain the N frames of first image frames.

10. A video image processing device, characterized in that: The device comprises: An input module, used for acquiring N first image frames, and inputting the N first image frames into a first network module; A first network module is used to perform super-resolution reconstruction on each of the N first image frames to obtain N second image frames; A second network module is used to perform alignment and fusion processing on the N frames of second image frames to obtain a result frame; A selection module is used to select a first key frame from the N first image frames, where the first key frame is an image frame in the N first image frames; wherein the first key frame is reconstructed by super-resolution through the first network module to obtain a second key frame, where the second key frame is an image frame in the N second image frames; The second network module is specifically used to use the second network module to align and fuse each second image frame in N-1 frames of second image frames with the second key frame to obtain N-1 first fused frames; the N-1 frames of second image frames are image frames other than the second key frame in the N frames of second image frames; use the second network module to align and fuse every two first fused frames in the N-1 first fused frames to obtain M second fused frames; M is an integer greater than or equal to 1 and less than N-1; and obtain a result frame based on the M second fused frames.

11. An electronic device, comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images