Image processing method and device and storage medium
Through shallow feature extraction, frame alignment processing and residual reconstruction of video data, the problem that the video super-resolution reconstruction method in the prior art cannot effectively combine space-time information, and high-quality high-resolution video generation is achieved, which improves the Reconstruct efficiency and accuracy.
Patent Information
- Application Number
- CN202310748973.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-21
- Publication Date
- 2024-12-24
AI Technical Summary
The existing video super-resolution reconstruction method cannot effectively combine space-time information, resulting in the reconstruction of video quality failing to meet the preset requirements, and there are serious distortion problems.
By acquiring video data, performing shallow feature extraction, frame alignment processing and residual reconstruction, high-resolution video data are generated, and the efficiency and accuracy of video super-resolution reconstruction are improved.
Accurate super-resolution reconstruction of low-resolution videos is achieved, and high-quality high-resolution videos are generated, greatly improving the efficiency and accuracy of video super-resolution reconstruction.
Smart Images
Figure CN119181037A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to an image processing method, device and storage medium. Background Art
[0002] High-resolution video has always been a goal pursued in daily life. For example, in the field of medical imaging, high-definition video streams are conducive to more accurate analysis and diagnosis; in the field of traffic monitoring, high-definition road monitoring videos are conducive to more accurate analysis and detection of traffic volume and traffic density; in the field of criminal investigation, high-definition videos are conducive to assisting the police in solving cases and arresting criminal suspects. Image super-resolution reconstruction (SR) is an image processing technology that uses computers to process a low-resolution image (LR) or image sequence to restore a high-resolution image (HR). Its core idea is to use temporal bandwidth to exchange for spatial resolution, and realize the conversion of temporal resolution to spatial resolution to obtain high-resolution video.
[0003] At present, video super-resolution methods are used for video super-resolution reconstruction tasks, which can be generally divided into traditional methods and deep learning methods. These two methods either only use the spatial or temporal correlation in the video, or cannot effectively combine spatiotemporal information to complete the video reconstruction task, resulting in the reconstructed video failing to meet the preset requirements or having serious distortion problems. Therefore, how to perform video super-resolution reconstruction on low-resolution videos to generate high-quality high-resolution videos is an urgent problem to be solved. Summary of the invention
[0004] The embodiments of the present invention provide an image processing method, device and storage medium, which are intended to perform video super-resolution reconstruction on low-resolution videos to generate high-quality high-resolution videos.
[0005] In a first aspect, an embodiment of the present invention provides an image processing method, including:
[0006] Acquire first video data, and perform shallow feature extraction on the first video data to generate multiple frames of first feature maps;
[0007] Performing frame alignment processing on multiple frames of the first feature maps to generate multiple frames of second feature maps, and performing residual reconstruction on the multiple frames of the first feature maps and the multiple frames of the second feature maps to generate multiple frames of third feature maps;
[0008] Features are added to the upsampled feature map of the first video data and multiple frames of the third feature map to generate second video data, and the resolution of the first video is smaller than the resolution of the second video data.
[0009] In a second aspect, an embodiment of the present invention further provides a terminal device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of any one of the image processing methods provided in the specification of the present invention are implemented.
[0010] In a third aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any image processing method provided in the specification of the present invention.
[0011] The embodiment of the present invention provides an image processing method, device and storage medium. The embodiment of the present invention obtains first video data, performs shallow feature extraction on the first video data, and generates multiple frames of first feature maps; performs frame alignment processing on the multiple frames of first feature maps to generate multiple frames of second feature maps, and performs residual reconstruction on the multiple frames of first feature maps and the multiple frames of second feature maps to generate multiple frames of third feature maps; and performs feature addition on the feature map after upsampling the first video data and the multiple frames of the third feature map, so as to accurately generate the second video data, and the resolution of the first video is less than the resolution of the second video data. This scheme performs shallow feature extraction, frame alignment processing and residual reconstruction processing on the low-resolution first video data, and can accurately obtain high-resolution second video data, which greatly improves the efficiency and accuracy of super-resolution reconstruction of low-resolution video to generate high-quality high-resolution video. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 A schematic diagram of a flow chart of an image processing method provided by an embodiment of the present invention;
[0013] Figure 2 for Figure 1 A schematic diagram of the sub-step flow chart of the image processing method in FIG.
[0014] Figure 3 A schematic diagram of a scene of an image processing method provided by an embodiment of the present invention;
[0015] Figure 4 for Figure 1 A schematic diagram of the sub-step flow chart of the image processing method in FIG.
[0016] Figure 5 A schematic diagram of a scene for frame alignment processing is provided for an embodiment of the present invention;
[0017] Figure 6A schematic diagram of another scenario of the image processing method provided by an embodiment of the present invention;
[0018] Figure 7 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0020] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0021] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0022] The embodiment of the present invention provides an image processing method, device and storage medium. The image processing method can be applied to a terminal device, which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer and other devices. For example, when the terminal device is a desktop computer, the desktop computer obtains the first video data, and performs shallow feature extraction on the first video data to generate multiple frames of first feature maps; performs frame alignment processing on the multiple frames of first feature maps to generate multiple frames of second feature maps, and performs residual reconstruction on the multiple frames of first feature maps and the multiple frames of second feature maps to generate multiple frames of third feature maps; performs feature addition on the feature map after upsampling the first video data and the multiple frames of the third feature map to generate the second video data, and the resolution of the first video is less than the resolution of the second video data.
[0023] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0024] Please refer to Figure 1 , Figure 1 A flowchart of an image processing method provided by an embodiment of the present invention.
[0025] like Figure 1As shown, the image processing method includes steps S101 to S103.
[0026] Step S101: Acquire first video data, and perform shallow feature extraction on the first video data to generate multiple frames of first feature maps.
[0027] The first video data and the second video data are compared in resolution, the first video data is a low-resolution video, and the second video data is a high-resolution video. For example, the resolution of the first video data is 112*64, and after super-resolution reconstruction of the first video data, the resolution of the second video data of the high-resolution video is 448*256.
[0028] In one embodiment, first video data is obtained, and the first video data includes multiple image frames. The number of image frames in the first video data is at least greater than a preset number. The preset number can be set according to actual conditions, and the embodiment of the present invention does not specifically limit this. For example, the preset number can be 7.
[0029] In one embodiment, if Figure 2 As shown, step S101 includes sub-steps S1011 to S1012.
[0030] Sub-step S1011 , pre-processing the first video data to generate target first video data.
[0031] The preprocessing includes at least one of the following: image frame size cropping, image frame flipping and image frame format conversion.
[0032] In one embodiment, each frame image in the first video data is processed by frame division to generate multiple frames of images, and each image frame is cropped, flipped, and converted to a different format to generate pre-processed first video data. By pre-processing the first video data, the efficiency and accuracy of super-resolution reconstruction can be improved.
[0033] Exemplarily, the first video data is a 112×64×3 video frame, wherein 112 represents the number of pixels in the horizontal direction of the image frame, 64 represents the number of pixels in the vertical direction of the image frame, and 3 represents the number of color channels of the image frame; the 112×64×3 video frame is cropped to generate a 64×64×3 video frame; the 64×64×3 video frame is flipped at an angle of 90°, 180°, or 270°, and converted to Lightning Memory-Mapped Database (lmdb) format to generate the target first video data.
[0034] Sub-step S1012: input the target first video data into a preset residual convolution model for shallow feature extraction to generate a multi-frame first feature map.
[0035] Wherein, the preset residual convolution model includes a plurality of residual convolution blocks, each residual convolution block includes a convolution block and an activation function. Wherein, the number of the residual convolution blocks can be adjusted according to actual conditions, and the embodiment of the present invention does not make specific limitations on this. The activation function can be selected according to actual conditions, and the embodiment of the present invention does not make specific limitations on this. For example, the activation function can be a Leaky ReLU activation function. Wherein, the shallow feature extraction is to convert each feature in the target first video data into a feature vector, and the feature includes but is not limited to features such as image frame resolution, color data, texture data, and water chestnut data.
[0036] For example, Figure 3 As shown, the residual convolution model 10 includes a residual convolution block 21 and a residual convolution block 22, wherein a plurality of residual convolution blocks are omitted between the residual convolution block 21 and the residual convolution block 22; the residual convolution block 21 includes a convolution block 31, an activation function 32 and a convolution block 33; the residual convolution block 22 includes a convolution block 34, an activation function 35 and a convolution block 36; the target first video data is sequentially input into the convolution block 31, the activation function 32 and the convolution block 33 of the residual convolution block 21, as well as the plurality of residual convolution blocks omitted in the middle, and finally input into the convolution block 34, the activation function 35 and the convolution block 36 in the residual convolution block 22 for processing, and outputs a plurality of frames of first feature maps.
[0037] Step S102: perform frame alignment processing on multiple frames of the first feature map to generate multiple frames of second feature maps, and perform residual reconstruction on multiple frames of the first feature map and multiple frames of the second feature map to generate multiple frames of third feature maps.
[0038] The second feature maps of the multiple frames are obtained by reconstructing the feature maps of the first feature maps of the multiple frames.
[0039] In one embodiment, if Figure 4 As shown, step S102 includes sub-steps S1021 and S1022.
[0040] Sub-step S1021, determining the reference frame feature map corresponding to the first feature map of each frame.
[0041] The reference frame feature map is a feature map of a previous frame or a feature map of a subsequent frame of the first feature map.
[0042] In one embodiment, when the first feature map of the current frame is not the last frame feature map among the first feature maps of multiple frames, the next frame feature map of the first feature map of the current frame is determined as the reference frame feature map of the first feature map of the current frame; when the first feature map of the current frame is the last frame feature map among the first feature maps of multiple frames, the previous frame feature map of the first feature map of the current frame is determined as the reference frame feature map of the first feature map of the current frame.
[0043] Exemplarily, the first video data includes, in the time order of frame acquisition, a first frame feature map, a second frame feature map, a third frame feature map, a fourth frame feature map, a fifth frame feature map, a sixth frame feature map and a seventh frame feature map, and the second frame feature map is used as a reference frame feature map of the first frame feature map, the third frame feature map is used as a reference frame feature map of the second frame feature map, the fourth frame feature map is used as a reference frame feature map of the third frame feature map, the fifth frame feature map is used as a reference frame feature map of the fourth frame feature map, the sixth frame feature map is used as a reference frame feature map of the fifth frame feature map, the seventh frame feature map is used as a reference frame feature map of the sixth frame feature map, and the sixth frame feature map is used as a reference frame feature map of the seventh frame feature map.
[0044] Sub-step S1022: performing frame alignment processing according to the first feature map of each frame and the corresponding reference frame feature map to generate a second feature map of each frame.
[0045] In one embodiment, the first feature map and the corresponding reference frame feature map are downsampled multiple times with the same scale factor to generate a first frame feature map, a first reference frame feature map, a second frame feature map, a second reference frame feature map, a third frame feature map, and a third reference frame feature map; the first frame feature map, the first reference frame feature map, the second frame feature map, the second reference frame feature map, the third frame feature map, and the third reference frame feature map are frame aligned according to a preset pyramid cascade structure to generate a second feature map. By downsampling the first feature map and the corresponding reference frame feature map, the accuracy of frame alignment can be improved.
[0046] Exemplarily, the size of the first feature map and the corresponding reference frame feature map is (T, C, H, W), where T is the number of feature map frames, C is the number of color channels of the feature map, H is the height of the feature map, and W is the width of the feature map; the first feature map and the corresponding reference frame feature map are downsampled with a scale factor of 4 to generate a first frame feature map and a first reference frame feature map with a size of (T, C, H / / 4, W / / 4); the second feature map and the corresponding reference frame feature map are downsampled with a scale factor of 2 to generate a second frame feature map and a second reference frame feature map with a size of (T, C, H / / 2, W / / 2); the first feature map and the corresponding reference frame feature map are downsampled with a scale factor of 1 to generate a third frame feature map and a third reference frame feature map with a size of (T, C, H, W).
[0047] In one embodiment, frame alignment processing is performed on the first frame feature map, the first reference frame feature map, the second frame feature map, the second reference frame feature map, the third frame feature map and the third reference frame feature map according to a preset pyramid cascade structure to generate the second feature map: the preset pyramid cascade structure includes a first hierarchical structure, a second hierarchical structure and a third hierarchical structure; the first frame feature map and the first reference frame feature map are input into the first hierarchical structure for frame alignment processing to generate a fourth frame feature map and a fourth reference frame feature map; the second frame feature map, the second reference frame feature map and the fourth reference frame feature map are input into the second hierarchical structure for frame alignment processing to generate a fifth frame feature map and a fifth reference frame feature map; the third frame feature map, the third reference frame feature map and the fifth reference frame feature map are input into the third hierarchical structure for frame alignment processing to generate a sixth frame feature map and a sixth reference frame feature map; the fourth frame feature map, the fifth frame feature map, the sixth frame feature map and the sixth reference frame feature map are fused to generate the second feature map. By performing frame alignment processing on the first frame feature map, the first reference frame feature map, the second frame feature map, the second reference frame feature map, the third frame feature map and the third reference frame feature map according to a preset pyramid cascade structure, the second feature map can be accurately obtained, which greatly improves the efficiency and accuracy of super-resolution reconstruction.
[0048] It should be noted that the preset pyramid cascade structure is a convolutional neural network model constructed based on preset convolutional layers and activation functions. The preset pyramid cascade structure can be designed according to actual conditions, and the embodiment of the present invention does not specifically limit this.
[0049] For example, Figure 5 As shown, Figure 5In order to perform frame alignment processing according to the first feature map of each frame and the corresponding reference frame feature map, a result schematic diagram of the second feature map of each frame is generated. The specific implementation process in the schematic diagram is as described in the steps of the following embodiments, wherein block diagram 41 is a first hierarchical structure, block diagram 42 is a second hierarchical structure, and block diagram 43 is a third hierarchical structure.
[0050] In one embodiment, the first frame feature map and the first reference frame feature map are input into the first hierarchical structure for frame alignment processing, and the fourth frame feature map and the fourth reference frame feature map are generated by: performing feature alignment operation on the first frame feature map and the first reference frame feature map to generate a first alignment feature map; performing correlation calculation on the first frame feature map and the first reference frame feature map through a preset graph neural network to generate a first similarity feature map; performing convolution multiplication on the first alignment feature map and the first similarity feature map to generate a fourth frame feature map and a fourth reference frame feature map. Among them, the preset graph neural network is a pre-trained neural network model, and the graph neural network can be trained according to actual conditions, and the embodiment of the present invention does not specifically limit this. By performing frame alignment processing on the first frame feature map and the first reference frame feature map through the first hierarchical structure, the fourth frame feature map and the fourth reference frame feature map can be accurately obtained.
[0051] In one embodiment, a feature alignment operation is performed on a first frame feature map and a first reference frame feature map to generate a first alignment feature map. The first alignment feature map includes an alignment feature map corresponding to the first frame feature map and an alignment feature map corresponding to the first reference frame feature map. Feature connection processing is performed on the first frame feature map and the first reference frame feature map to generate a first frame fusion feature map. Convolution transformation and offset convolution are performed on the first frame fusion feature map to obtain a first offset between the first frame feature map and the first reference frame feature map. A preset deformable convolution is performed to perform a feature alignment operation on the first frame feature map and the first offset to generate an alignment feature map corresponding to the first frame feature map. A preset deformable convolution is performed to perform a feature alignment operation on the first reference frame feature map and the first offset to generate an alignment feature map corresponding to the first reference frame feature map.
[0052] In one embodiment, the first alignment feature map and the first similarity feature map are convolved and multiplied to generate a fourth frame feature map and a fourth reference frame feature map. The method can be as follows: the alignment feature map corresponding to the first frame feature map and the first similarity feature map are convolved and multiplied to generate the fourth frame feature map; the alignment feature map corresponding to the first reference frame feature map and the first similarity feature map are convolved and multiplied and upsampled to generate the fourth reference frame feature map.
[0053] In one embodiment, the second frame feature map, the second reference frame feature map and the fourth reference frame feature map are input into the second hierarchical structure for frame alignment processing, and the fifth frame feature map and the fifth reference frame feature map are generated by: performing feature alignment operation on the second frame feature map and the second reference frame feature map to generate a second aligned feature map; performing correlation calculation on the second frame feature map and the second reference frame feature map through a preset graph neural network to generate a second similar feature map; and generating a fifth frame feature map and a fifth reference frame feature map based on the second aligned feature map, the second similar feature map and the fourth reference frame feature map. By performing frame alignment processing on the second frame feature map, the second reference frame feature map and the fourth reference frame feature map through the second hierarchical structure, the fifth frame feature map and the fifth reference frame feature map can be accurately obtained.
[0054] In one embodiment, a feature alignment operation is performed on the second frame feature map and the second reference frame feature map to generate the second aligned feature map in the following manner: the second aligned feature map includes an aligned feature map corresponding to the second frame feature map and an aligned feature map corresponding to the second reference frame feature map; feature connection processing is performed on the second frame feature map and the second reference frame feature map to generate a second fused feature map; the first offset is upsampled to generate a target first offset; convolution transformation and offset convolution are performed on the second fused feature map and the target first offset to obtain a second offset of the second frame feature map and the second reference frame feature map; a preset deformable convolution is performed on the second frame feature map and the second offset to generate an aligned feature map corresponding to the second frame feature map; a preset deformable convolution is performed on the second reference frame feature map and the second offset to generate an aligned feature map corresponding to the second reference frame feature map.
[0055] In one embodiment, the method of generating the fifth frame feature map and the fifth reference frame feature map according to the second alignment feature map, the second similarity feature map and the fourth reference frame feature map can be: performing convolution multiplication on the alignment feature map and the second similarity feature map corresponding to the second frame feature map to generate the fifth frame feature map; performing convolution multiplication on the alignment feature map and the second similarity feature map corresponding to the second reference frame feature map, and then merging them with the fourth reference frame feature map, performing activation function and convolution processing to generate the fifth reference frame feature map.
[0056] In one embodiment, the third frame feature map, the third reference frame feature map and the fifth reference frame feature map are input into the third hierarchical structure for frame alignment processing, and the sixth frame feature map and the sixth reference frame feature map are generated in the following manner: feature alignment operation is performed on the third frame feature map and the third reference frame feature map to generate a third alignment feature map; correlation calculation is performed on the third frame feature map and the third reference frame feature map through a preset graph neural network to generate a third similarity feature map; and the sixth frame feature map and the sixth reference frame feature map are generated according to the third alignment feature map, the third similarity feature map and the fifth reference frame feature map. By performing frame alignment processing on the third frame feature map, the third reference frame feature map and the fifth reference frame feature map through the third hierarchical structure, the sixth frame feature map and the sixth reference frame feature map can be accurately obtained.
[0057] In one embodiment, a feature alignment operation is performed on the third frame feature map and the third reference frame feature map to generate a third aligned feature map in the following manner: the third aligned feature map includes an aligned feature map corresponding to the third frame feature map and an aligned feature map corresponding to the third reference frame feature map; feature connection processing is performed on the third frame feature map and the third reference frame feature map to generate a third fused feature map; the second offset is upsampled to generate a target second offset; convolution transformation and offset convolution are performed on the third fused feature map and the target second offset to obtain a third offset of the third frame feature map and the third reference frame feature map; a preset deformable convolution is performed on the third frame feature map and the third offset to generate an aligned feature map corresponding to the third frame feature map; a preset deformable convolution is performed on the third reference frame feature map and the third offset to generate an aligned feature map corresponding to the third reference frame feature map.
[0058] In one embodiment, the method of generating the sixth frame feature map and the sixth reference frame feature map according to the third alignment feature map, the third similarity feature map and the fifth reference frame feature map can be: performing convolution multiplication on the alignment feature map and the third similarity feature map corresponding to the third frame feature map to generate the sixth frame feature map; performing convolution multiplication on the alignment feature map and the third similarity feature map corresponding to the third reference frame feature map, and then merging them with the fifth reference frame feature map, performing activation function and convolution processing, to generate the sixth reference frame feature map.
[0059] In one embodiment, the fourth frame feature map, the fifth frame feature map, the sixth frame feature map and the sixth reference frame feature map are fused to generate the second feature map in the following manner: upsampling the fourth frame feature map to generate the seventh frame feature map, merging, convolving and performing activation function processing on the fifth frame feature map and the seventh frame feature map to generate the eighth frame feature map; upsampling the eighth frame feature map to generate the ninth frame feature map; merging, convolving and performing activation function processing on the sixth frame feature map and the sixth reference frame feature to generate the sixth fused feature map; merging, activating and fusion processing on the sixth fused feature map and the ninth frame feature map to generate the second feature map.
[0060] In one embodiment, feature fusion processing is performed on multiple frames of first feature maps and multiple frames of second feature maps to generate multiple frames of fourth feature maps, and upsampling processing is performed on multiple frames of fourth feature maps to generate multiple frames of fifth feature maps; multiple frames of fifth feature maps are processed by a preset residual block and a preset convolutional layer to generate multiple frames of third feature maps. The preset residual block and the preset convolutional layer are pre-trained. By performing feature fusion processing on multiple frames of first feature maps and multiple frames of second feature maps, multiple frames of fourth feature maps can be accurately generated.
[0061] Step S103, adding features of the upsampled feature map of the first video data and multiple frames of the third feature map to generate second video data, wherein the resolution of the first video is smaller than the resolution of the second video data.
[0062] The first video data is upsampled to generate multiple frames of sixth feature maps, the multiple frames of sixth feature maps and the multiple frames of third feature maps are feature added to generate multiple frames of target feature maps, and the multiple frames of target feature maps are converted into video data to generate second video data. By adding the feature maps of the upsampled first video data and the multiple frames of third feature maps, the overfitting phenomenon of feature reconstruction can be avoided, and the accuracy of super-resolution reconstruction is greatly improved.
[0063] For example, Figure 6 As shown, multiple frames of first feature maps 52 and multiple frames of second feature maps 53 are input into a residual model 54, and feature fusion, upsampling, residual processing and convolution processing are performed to generate multiple frames of third feature maps. The first video data is upsampled 55 to generate multiple frames of sixth feature maps. Features of the multiple frames of sixth feature maps and the multiple frames of third feature maps are added to generate second video data.
[0064] It should be noted that a plurality of sample data are obtained, the sample data including sample first video data and annotated second video data; shallow feature extraction is performed on the sample first video data to generate a plurality of first feature maps, the resolution of the first video is less than or equal to the preset resolution; frame alignment is performed on the plurality of first feature maps to generate a plurality of second feature maps, and residual reconstruction is performed on the plurality of first feature maps and the plurality of second feature maps to generate a plurality of third feature maps; feature addition is performed on the feature map after upsampling the first video data and the plurality of third feature maps to generate predicted second video data, based on the predicted second video data and the annotated second video data, it is determined whether each convolution block and residual block converge in the process of obtaining the predicted second video data, and if each convolution block and residual block do not converge, the model parameters of each residual block and convolution block are updated until convergence. It should be noted that the training process of each convolution block and residual block can refer to the training process of the neural network model, and the embodiment of the present invention does not specifically limit this.
[0065] The image processing method in the above embodiment acquires the first video data, performs shallow feature extraction on the first video data, and generates multiple frames of first feature maps; performs frame alignment processing on the multiple frames of first feature maps to generate multiple frames of second feature maps, and performs residual reconstruction on the multiple frames of first feature maps and the multiple frames of second feature maps to generate multiple frames of third feature maps; performs feature addition on the feature map after upsampling the first video data and the multiple frames of the third feature map, and can accurately generate the second video data, and the resolution of the first video is less than the resolution of the second video data. This solution performs shallow feature extraction, frame alignment processing, and residual reconstruction processing on the low-resolution first video data, and can accurately obtain high-resolution second video data, which greatly improves the efficiency and accuracy of super-resolution reconstruction of low-resolution video to generate high-quality high-resolution video.
[0066] See also Figure 7 , Figure 7 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention.
[0067] like Figure 7 As shown, the terminal device 600 includes a processor 601 and a memory 602, and the processor 601 and the memory 602 are connected via a bus 603, such as an I2C (Inter-integrated Circuit) bus.
[0068] Specifically, the processor 601 is used to provide computing and control capabilities to support the operation of the entire terminal device. The processor 601 can be a central processing unit (CPU), and the processor 601 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.
[0069] Specifically, the memory 602 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.
[0070] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present invention, and does not constitute a limitation on the terminal device to which the scheme of the present invention is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0071] The processor is used to run a computer program stored in the memory, and implement any one of the image processing methods provided by the embodiments of the present invention when executing the computer program.
[0072] In one embodiment, the processor is used to run a computer program stored in the memory, and implements the following steps when executing the computer program:
[0073] Acquire first video data, and perform shallow feature extraction on the first video data to generate multiple frames of first feature maps;
[0074] Performing frame alignment processing on multiple frames of the first feature maps to generate multiple frames of second feature maps, and performing residual reconstruction on the multiple frames of the first feature maps and the multiple frames of the second feature maps to generate multiple frames of third feature maps;
[0075] Features are added to the upsampled feature map of the first video data and multiple frames of the third feature map to generate second video data, and the resolution of the first video is smaller than the resolution of the second video data.
[0076] In one embodiment, when the processor performs shallow feature extraction on the first video data to generate multiple frames of first feature maps, it is used to implement:
[0077] Preprocessing the first video data to generate target first video data;
[0078] The target first video data is input into a preset residual convolution model for shallow feature extraction to generate a multi-frame first feature map.
[0079] In one embodiment, the preprocessing includes at least one of the following: image frame size cropping, image frame flipping, and image frame format conversion.
[0080] In one embodiment, when the processor performs frame alignment processing on multiple frames of the first feature map to generate multiple frames of the second feature map, it is used to implement:
[0081] Determine a reference frame feature map corresponding to the first feature map of each frame;
[0082] Frame alignment is performed based on the first feature map of each frame and the corresponding reference frame feature map to generate a second feature map of each frame.
[0083] In one embodiment, when the processor implements the step of determining the reference frame feature map corresponding to the first feature map of each frame, the processor is used to implement:
[0084] When the first feature map of the current frame is not the last feature map of the first feature maps of multiple frames, determining the next frame feature map of the first feature map of the current frame as the reference frame feature map of the first feature map of the current frame;
[0085] In the case where the first feature map of the current frame is the last frame feature map among the first feature maps of multiple frames, the feature map of the previous frame of the first feature map of the current frame is determined as the reference frame feature map of the first feature map of the current frame.
[0086] In one embodiment, when the processor performs frame alignment processing according to the first feature map of each frame and the corresponding reference frame feature map to generate the second feature map of each frame, it is used to implement:
[0087] Downsampling the first feature map and the corresponding reference frame feature map at the same scale factor for multiple times to generate a first frame feature map, a first reference frame feature map, a second frame feature map, a second reference frame feature map, a third frame feature map and a third reference frame feature map;
[0088] The first frame feature map, the first reference frame feature map, the second frame feature map, the second reference frame feature map, the third frame feature map and the third reference frame feature map are subjected to frame alignment processing according to a preset pyramid cascade structure to generate a second feature map.
[0089] In one embodiment, the processor implements the preset pyramid cascade structure including a first hierarchical structure, a second hierarchical structure, and a third hierarchical structure; the first frame feature map, the first reference frame feature map, the second frame feature map, the second reference frame feature map, the third frame feature map, and the third reference frame feature map are subjected to frame alignment processing according to the preset pyramid cascade structure to generate the second feature map, and is used to implement:
[0090] Inputting the first frame feature map and the first reference frame feature map into the first hierarchical structure for frame alignment processing to generate a fourth frame feature map and a fourth reference frame feature map;
[0091] Inputting the second frame feature map, the second reference frame feature map and the fourth reference frame feature map into the second hierarchical structure for frame alignment processing to generate a fifth frame feature map and a fifth reference frame feature map;
[0092] Inputting the third frame feature map, the third reference frame feature map and the fifth reference frame feature map into the third hierarchical structure for frame alignment processing to generate a sixth frame feature map and a sixth reference frame feature map;
[0093] The fourth frame feature map, the fifth frame feature map, the sixth frame feature map and the sixth reference frame feature map are fused to generate the second feature map.
[0094] In one embodiment, when the processor implements the step of inputting the first frame feature map and the first reference frame feature map into the first hierarchical structure for frame alignment processing to generate a fourth frame feature map and a fourth reference frame feature map, it is configured to implement:
[0095] Performing a feature alignment operation on the first frame feature map and the first reference frame feature map to generate a first aligned feature map;
[0096] Performing correlation calculation on the first frame feature map and the first reference frame feature map through a preset graph neural network to generate a first similarity feature map;
[0097] The first alignment feature map and the first similarity feature map are convolved and multiplied to generate the fourth frame feature map and the fourth reference frame feature map.
[0098] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned image processing method embodiment, and will not be repeated here.
[0099] An embodiment of the present invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any image processing method provided in the specification of the present invention.
[0100] The storage medium may be an internal storage unit of the terminal device described in the foregoing embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc., equipped on the terminal device.
[0101] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0102] It should be understood that the term "and / or" used in the present specification refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.
[0103] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.
Claims
1. An image processing method, characterized in that: include: Acquire first video data, and perform shallow feature extraction on the first video data to generate multiple frames of first feature maps; Performing frame alignment processing on multiple frames of the first feature maps to generate multiple frames of second feature maps, and performing residual reconstruction on the multiple frames of the first feature maps and the multiple frames of the second feature maps to generate multiple frames of third feature maps; Features are added to the upsampled feature map of the first video data and multiple frames of the third feature map to generate second video data, and the resolution of the first video is smaller than the resolution of the second video data.
2. The image processing method according to claim 1, characterized in that: The performing shallow feature extraction on the first video data to generate multiple frames of first feature maps includes: Preprocessing the first video data to generate target first video data; The target first video data is input into a preset residual convolution model for shallow feature extraction to generate a multi-frame first feature map.
3. The image processing method according to claim 2, characterized in that: The preprocessing includes at least one of the following: image frame size cropping, image frame flipping and image frame format conversion.
4. The image processing method according to claim 1, characterized in that: The performing frame alignment processing on the multiple frames of the first feature maps to generate multiple frames of the second feature maps includes: Determine a reference frame feature map corresponding to the first feature map of each frame; Frame alignment is performed based on the first feature map of each frame and the corresponding reference frame feature map to generate a second feature map of each frame.
5. The image processing method according to claim 4, characterized in that: The determining of the reference frame feature map corresponding to the first feature map of each frame includes: When the first feature map of the current frame is not the last feature map of the first feature maps of multiple frames, determining the next frame feature map of the first feature map of the current frame as the reference frame feature map of the first feature map of the current frame; In the case where the first feature map of the current frame is the last frame feature map among the first feature maps of multiple frames, the feature map of the previous frame of the first feature map of the current frame is determined as the reference frame feature map of the first feature map of the current frame.
6. The image processing method according to claim 4, characterized in that: The performing frame alignment processing according to the first feature map of each frame and the corresponding reference frame feature map to generate the second feature map of each frame includes: Downsampling the first feature map and the corresponding reference frame feature map at the same scale factor for multiple times to generate a first frame feature map, a first reference frame feature map, a second frame feature map, a second reference frame feature map, a third frame feature map and a third reference frame feature map; The first frame feature map, the first reference frame feature map, the second frame feature map, the second reference frame feature map, the third frame feature map and the third reference frame feature map are subjected to frame alignment processing according to a preset pyramid cascade structure to generate a second feature map.
7. The image processing method according to claim 6, characterized in that: The preset pyramid cascade structure includes a first hierarchical structure, a second hierarchical structure, and a third hierarchical structure; the frame alignment processing is performed on the first frame feature map, the first reference frame feature map, the second frame feature map, the second reference frame feature map, the third frame feature map, and the third reference frame feature map according to the preset pyramid cascade structure to generate a second feature map, including: Inputting the first frame feature map and the first reference frame feature map into the first hierarchical structure for frame alignment processing to generate a fourth frame feature map and a fourth reference frame feature map; Inputting the second frame feature map, the second reference frame feature map and the fourth reference frame feature map into the second hierarchical structure for frame alignment processing to generate a fifth frame feature map and a fifth reference frame feature map; Inputting the third frame feature map, the third reference frame feature map and the fifth reference frame feature map into the third hierarchical structure for frame alignment processing to generate a sixth frame feature map and a sixth reference frame feature map; The fourth frame feature map, the fifth frame feature map, the sixth frame feature map and the sixth reference frame feature map are fused to generate the second feature map.
8. The image processing method according to claim 7, characterized in that: The step of inputting the first frame feature map and the first reference frame feature map into the first hierarchical structure for frame alignment processing to generate a fourth frame feature map and a fourth reference frame feature map includes: Performing a feature alignment operation on the first frame feature map and the first reference frame feature map to generate a first aligned feature map; Performing correlation calculation on the first frame feature map and the first reference frame feature map through a preset graph neural network to generate a first similarity feature map; The first alignment feature map and the first similarity feature map are convolved and multiplied to generate the fourth frame feature map and the fourth reference frame feature map.
9. A terminal device, characterized in that: The terminal device includes a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of the image processing method as described in any one of claims 1 to 8 are realized.
10. A storage medium for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the image processing method according to any one of claims 1 to 8.