Multi-frame super-resolution reconstruction method and device, equipment and storage medium

Abstract

The invention discloses a multi-frame super-resolution reconstruction method and device, equipment and a storage medium, and the method comprises the steps: carrying out the alignment processing of a plurality of frames of first images, and obtaining a plurality of frames of second images after alignment; performing feature extraction on the multiple frames of second images to obtain feature information of the multiple frames of second images; determining a fusion weight of the feature information based on a similarity weight between the feature information of the multiple frames of second images; based on the fusion weight of the feature information, performing similarity weighted fusion on the feature information of the multiple frames of second images to obtain fused feature information; and performing image reconstruction according to the fused feature information to obtain a super-resolution image. Therefore, multi-frame super-resolution reconstruction in the real world is realized by paying attention to the similarity of inter-frame and intra-frame information of the low-resolution multi-frame image and effectively integrating the sub-pixel detail information contained in the multi-frame image sequence, the reconstruction quality can be effectively improved, and the method has important significance on multi-frame super-resolution research in the real world.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to a multi-frame super-resolution reconstruction method and apparatus, device, and storage medium. Background Art

[0002] Image super-resolution (SR) refers to the process of restoring an image from low resolution to high resolution using an algorithm. It is one of the important technologies in computer vision and imaging. High resolution means that there are more pixels in the image and more details can be presented, which is very important for many practical applications. For example, high-resolution medical images can help doctors make correct diagnoses; high-resolution satellite images can make it easier for people to distinguish similar objects, and so on.

[0003] In recent years, super-resolution technology mainly includes single-frame super-resolution tasks and multi-frame super-resolution tasks. Single-frame super-resolution tasks are fundamentally limited by available information (single frame), so they can only restore lost detail information through learned image priors. In contrast, multi-frame super-resolution tasks can combine information from multiple low-resolution images to generate high-resolution image outputs. Compared with single-frame super-resolution tasks, multi-frame super-resolution tasks can use this rich additional information to generate higher quality images. However, multi-frame super-resolution tasks are not only restricted by data sets, but also have to overcome many difficulties when designing solutions. The difficulty of multi-frame super-resolution tasks lies in the alignment and fusion of multiple frames. Summary of the invention

[0004] The embodiments of the present application are intended to provide a multi-frame super-resolution reconstruction method and apparatus, device, and storage medium.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, a multi-frame super-resolution reconstruction method is provided, comprising:

[0007] Performing alignment processing on the multiple frames of first images to obtain the aligned multiple frames of second images;

[0008] Performing feature extraction on the multiple frames of second images to obtain feature information of the multiple frames of second images;

[0009] Determining a fusion weight of the feature information based on similarity weights between the feature information of the multiple frames of second images;

[0010] Based on the fusion weight of the feature information, similarity-weighted fusion is performed on the feature information of the multiple frames of the second image to obtain fused feature information;

[0011] Image reconstruction is performed according to the fused feature information to obtain a super-resolution image.

[0012] In a second aspect, a multi-frame super-resolution reconstruction device is provided, comprising: an alignment unit, a feature extraction unit, a joint similarity fusion unit, and a reconstruction unit;

[0013] The alignment unit is used to perform alignment processing on the multiple frames of first images to obtain the aligned multiple frames of second images;

[0014] The feature extraction unit is used to extract features from the multiple frames of second images to obtain feature information of the multiple frames of second images;

[0015] A joint similarity fusion unit is used to determine a fusion weight of the feature information based on the similarity weights between the feature information of the multiple frames of the second image; based on the fusion weights of the feature information, perform similarity weighted fusion on the feature information of the multiple frames of the second image to obtain fused feature information;

[0016] The reconstruction unit is used to perform image reconstruction according to the fused feature information to obtain a super-resolution image.

[0017] In a third aspect, a multi-frame super-resolution reconstruction device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor,

[0018] Wherein, the processor is configured to execute the steps of the aforementioned method when running the computer program.

[0019] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0020] In the embodiment of the present application, a multi-frame super-resolution reconstruction method and apparatus, device, and storage medium are provided, the method comprising: aligning multiple frames of first images to obtain aligned multiple frames of second images; extracting features from the multiple frames of second images to obtain feature information of the multiple frames of second images; determining the fusion weight of the feature information based on the similarity weights between the feature information of the multiple frames of second images; performing similarity-weighted fusion on the feature information of the multiple frames of second images based on the fusion weights of the feature information to obtain fused feature information; reconstructing images based on the fused feature information to obtain super-resolution images. In this way, by focusing on the similarity of inter-frame and intra-frame information of low-resolution multi-frame images, the sub-pixel detail information contained in the multi-frame image sequence is effectively integrated to achieve multi-frame super-resolution reconstruction of the real world, which can effectively improve the reconstruction quality and is of great significance to the research on multi-frame super-resolution of the real world. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1This is a schematic diagram of a first process flow of a multi-frame super-resolution reconstruction method in an embodiment of the present application;

[0022] Figure 2 The present invention provides a fusion method flow diagram for combining multiple frames with similarity guidance. Figure 1 ;

[0023] Figure 3 The present invention provides a fusion method flow diagram for combining multiple frames with similarity guidance. Figure 2 ;

[0024] Figure 4 The structure of the multi-resolution reconstruction model in the embodiment of the present application is shown in FIG. Figure 1 ;

[0025] Figure 5 The structure of the multi-resolution reconstruction model in the embodiment of the present application is shown in FIG. Figure 2 ;

[0026] Figure 6 Schematic diagram of the comparison of the fusion process of the FAF module and the VAF module in the embodiment of the present application;

[0027] Figure 7 Schematic diagram of the comparison of the fusion results of the FAF module and the VAF module in the embodiment of the present application;

[0028] Figure 8 This is a comparison of the results of various super-resolution models trained using RGB domain datasets in the embodiments of this application;

[0029] Fig. 9 Comparison of the results of various super-resolution models trained with RAW domain datasets in the embodiments of this application;

[0030] Fig.10 This is a schematic diagram of the composition structure of a multi-frame super-resolution reconstruction device in an embodiment of the present application;

[0031] Fig.11 Schematic diagram of the composition structure of a multi-frame super-resolution reconstruction device in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0033] Figure 1 Schematic diagram of the first process of the multi-frame super-resolution reconstruction method in the embodiment of the present application. Figure 1 As shown, the method may specifically include:

[0034] Step 101: performing alignment processing on multiple frames of first images to obtain multiple aligned frames of second images;

[0035] It should be noted that the first image can be a low-resolution original image. Aligning multiple frames of the first image can eliminate physical jitter between the multiple frames of the first image. The pixel displacement in the multiple-frame image sequence taken in rapid succession mainly comes from camera motion and scene changes. These pixel displacements are generally considered to be supplementary detail information that can be used in reconstruction. Before fusion, the multiple-frame image sequence needs to be aligned first to avoid artifacts or erroneous super-resolution results caused by information confusion or misalignment.

[0036] In some embodiments, the alignment method may be one of the following: optical flow, deformable, or homography.

[0037] Optionally, the embodiment of the present application uses a homography matrix to perform multi-frame alignment. Specifically, determine the homography matrix between the reference frame and other frames in the multi-frame first image; use the homography matrix to align the other frames with the reference frame to obtain the aligned multi-frame second image. It should be noted that the reference frame can be a preset frame in the multi-frame first image, for example, the reference frame is the first frame in the multi-frame first frame. The other frames are any frames in the multi-frame first image except the reference frame, and the homography matrices of the reference frame and other frames can be calculated in chronological order.

[0038] Step 102: extracting features from the multiple frames of second images to obtain feature information of the multiple frames of second images;

[0039] In some embodiments, feature extraction is performed on multiple frames of second images to obtain feature information of the multiple frames of second images, including: feature extraction is performed on multiple frames of second images through a convolutional neural network to obtain feature information of the multiple frames of second images for further feature fusion.

[0040] It should be noted that multiple convolutional neural networks can be used to extract features from multiple frames of low-resolution second images, that is, one convolutional neural network corresponds to one frame of the second image, to achieve higher feature extraction efficiency. It is also possible to extract features from the second images in sequence through a convolutional neural network. Exemplarily, the convolutional neural network includes an input layer, a convolutional layer, a pooling layer, an activation layer, a fully connected layer, and a softmax layer.

[0041] It should be noted that the feature information may include multi-channel features of the image, that is, include multiple feature maps (Feature Maps).

[0042] Step 103: determining a fusion weight of the feature information based on similarity weights between feature information of multiple frames of second images;

[0043] It should be noted that the similarity weight can be understood as a parameter that characterizes the similarity of image features. The similarity weight can specifically include the self-similarity weight of intra-frame features and the similarity weight of inter-frame features. The fusion weight is the weight parameter of the feature information of each frame when weighted fusion of multi-frame features is performed. Specifically, it can include the weight value of each element in the feature information. Weighted fusion based on the fusion weight can aggregate inter-frame and intra-frame information, and at the same time, weighted attention is paid to the complementary information between multiple frames, so as to obtain a more comprehensive weighted fusion result with richer information, thereby improving the quality of subsequent super-resolution reconstruction.

[0044] In some embodiments, based on the similarity weights between feature information of multiple frames of second images, determining the fusion weight of the feature information includes: taking the i-th frame in the multiple frames of second images as a reference frame, and determining the reference similarity weight according to the feature information of the i-th frame; determining the similarity weight between the feature information of the i-th frame and the feature information of the j-th frame according to the feature information of the i-th frame and the feature information of the j-th frame in the multiple frames of second images; wherein the value of j is not equal to i; determining the fusion weight of the feature information according to the reference similarity weight and the similarity weight between the feature information of the i-th frame and the feature information of the j-th frame.

[0045] It should be noted that the reference similarity weight is a reference value obtained based on the feature information of the reference frame to characterize the similarity between other frames and the reference frame. The closer the similarity weight of other frames is to the reference similarity weight, the more similar the other frames are to the reference frame.

[0046] Exemplarily, based on the feature information of the i-th frame, a reference similarity weight is determined, including: performing a dot product operation on the feature information of the i-th frame and the feature information of the i-th frame, and obtaining a similarity weight between the feature information of the i-th frame and the feature information of the i-th frame as the reference similarity weight; based on the feature information of the i-th frame and the feature information of the j-th frame in multiple frames of second images, a similarity weight between the feature information of the i-th frame and the feature information of the j-th frame is determined, including: performing a dot product operation on the feature information of the i-th frame and the feature information of the j-th frame in multiple frames of second images, and obtaining a similarity weight between the feature information of the i-th frame and the feature information of the j-th frame.

[0047] Exemplarily, the fusion weight of the feature information is determined according to the reference similarity weight and the similarity weight between the feature information of the i-th frame and the feature information of the j-th frame, including: taking the reference similarity weight as the fusion weight of the feature information of the i-th frame; performing a subtraction operation on the similarity weight between the feature information of the i-th frame and the feature information of the j-th frame and the reference similarity weight to obtain the fusion weight of the feature information of the j-th frame.

[0048] It should be noted that the fusion weight can be understood as a similarity difference between the feature information of the jth frame and the feature information of the ith frame. According to the weighted similarity difference between the low-resolution multi-frame image sequences, the complementary information between the multiple frames is emphasized, and the features of the multi-frame image sequences are weightedly fused to obtain a more comprehensive weighted fusion result with richer information.

[0049] In some embodiments, the i-th frame includes a preset frame in the multiple frames of the second image, that is, when performing feature fusion, the fusion weight between the feature information of any frame (including the reference frame) and the feature information of a reference frame is calculated to obtain a set of fusion weights. Exemplarily, the i-th frame is the first frame in the multiple frames of the second frame, which can also be called a base frame. That is to say, the first frame is used as a reference frame to calculate the fusion weight between a frame in the second image sequence and the first frame.

[0050] In some other embodiments, the i-th frame includes at least two frames preset in the multiple frames of the second image, that is, when performing feature fusion, the fusion weight between the feature information of any frame (including the reference frame) and the feature information of at least two reference frames is calculated to obtain at least two sets of fusion weights. Exemplarily, the i-th frame is any frame in the multiple frames of the second frame, that is, each frame is used as a reference frame to calculate the fusion weight between a certain frame and all other frames in the second image sequence.

[0051] Specifically, the implementation process of calculating the fusion weight can be expressed as:

[0052] A i,i =F i ·F i

[0053] A i,j =F i ·F j

[0054] D i =A i,i ,D j =d(A i,j ,A i,i ), when j≠i

[0055] Among them, F i is the feature information of the i-th frame, that is, the feature information of the reference frame, F j is the feature information of the jth frame, A i,j is the similarity weight between the feature information of the i-th frame and the feature information of the j-th frame, specifically, the dot product result of the feature information of the i-th frame and the feature information of the j-th frame, A i,i is the similarity weight between the feature information of the i-th frame and itself, that is, the reference similarity weight, D i is the fusion weight of the i-th frame, Dj is the fusion weight of the jth frame, and d() is the difference operation.

[0056] It should be noted that when performing super-resolution reconstruction of N frames, the value of i is between 1 and N, j≠i, and D i and D j Identify a set of fusion weights corresponding to the i-th frame as the reference frame.

[0057] For example, Figure 2 The present invention provides a fusion method flow diagram for combining multiple frames with similarity guidance. Figure 1 ,like Figure 2 As shown, when the i-th frame is the first frame, the implementation process of calculating the fusion weight can be expressed as:

[0058] A i,j =F i ·F j

[0059] D 1 =A 1,1 ,D j =d(A 1,j ,A 1,1 ), when j≠1

[0060] It should be noted that D 1 It can also be further expressed as D 1,1 , specifically used to indicate the fusion weight of the feature information of the first frame when the first frame is used as the reference frame, D j It can also be further expressed as D 1,j , specifically used to indicate the fusion weight of the feature information of the jth frame when the first frame is used as the reference frame.

[0061] Step 104: Based on the fusion weight of the feature information, perform similarity weighted fusion on the feature information of multiple frames of the second image to obtain fused feature information;

[0062] In some embodiments, based on the fusion weight of the feature information, similarity weighted fusion is performed on the feature information of multiple frames of the second image to obtain fused feature information, including: when the i-th frame is used as a reference frame, similarity weighted fusion is performed with the feature information of all frames according to the fusion weights of all frames to determine the fused feature information corresponding to the i-th frame as the reference frame. It should be noted that the feature information may include multiple feature maps, and the fusion weight of each frame is the fusion weight of multiple feature maps.

[0063] Exemplarily, similarity-weighted fusion is performed based on the fusion weights of all frames and the feature information of all frames to determine the fusion feature information corresponding to the i-th frame as the reference frame, including: performing a Hadamard product operation on the fusion weight of the k-th frame and the feature information of the k-th frame to obtain a Hadamard product operation result; wherein the value of k is equal to i or not equal to i; fusing the Hadamard product operation results corresponding to multiple frames of the second image to obtain the fusion feature information corresponding to the i-th frame as the reference frame. It should be noted that the fusion can be an addition operation (add) or a concatenation operation (concat). Optionally, the Dammar product operation results of the elements in the feature map corresponding to the Dammar product operation results are accumulated to increase the data volume of the feature map elements.

[0064] Specifically, when performing N-frame super-resolution reconstruction, the fusion implementation process can be expressed as:

[0065]

[0066] Among them, F i is the feature information of the i-th frame, that is, the feature information of the reference frame, F j is the feature information of the jth frame, D i is the fusion weight of the i-th frame, D j is the fusion weight of the jth frame, N is the number of low-resolution images involved in super-resolution reconstruction, M is the feature after the final fusion of the low-resolution multi-frame image sequence, and ° is the Hadamard product.

[0067] Furthermore, the i-th frame includes at least two frames preset in the multiple frames of the second image; based on the fusion weight of the feature information, the feature information of the multiple frames of the second image is similarity-weighted fused to obtain fused feature information, and further includes: fusing at least two groups of fused feature information corresponding to the at least two preset frames as reference frames to obtain final fused feature information. It should be noted that the fusion can be an accumulation operation or a splicing operation. Optionally, the at least two groups of fused features are spliced ​​to retain more feature information.

[0068] Specifically, when performing N-frame super-resolution reconstruction, the fusion implementation process can be expressed as:

[0069]

[0070] Among them, F i is the feature information of the i-th frame, that is, the feature information of the reference frame, F j is the feature information of the jth frame, D i is the fusion weight of the i-th frame, D jis the fusion weight of the jth frame, K is the number of reference frames, the value of K is greater than or equal to 1 and less than or equal to N, and N is the number of low-resolution images involved in super-resolution reconstruction. M is the feature after the final low-resolution multi-frame image sequence is fused, and ° is the Hadamard product.

[0071] For example, Figure 3 The present invention provides a fusion method flow diagram for combining multiple frames with similarity guidance. Figure 2 ,like Figure 3 As shown, when N frames of super-resolution reconstruction are performed, the fusion implementation process can also be expressed as:

[0072]

[0073] Among them, F i is the feature information of the i-th frame, that is, the feature information of the reference frame, F j is the feature information of the jth frame, A i,i is the similarity weight between the feature information of the i-th frame and itself, that is, the reference similarity weight, A i,j is the similarity weight between the feature information of the i-th frame and the feature information of the j-th frame, D i is the fusion weight of the i-th frame, D j is the fusion weight of the jth frame, d() is the difference operation. M is the feature after the final low-resolution multi-frame image sequence is fused, and ° is the Hadamard product. A i,i °F i It is a self-similarity feature, which can alleviate the problem that some fusion methods over-emphasize the fusion of redundant information similar to the basic frame and avoid the negative impact of the reconstructed area with high similarity (such as the plane area); Fusion of weighted features for inter-frame similarity will mitigate the adverse effects of misalignment due to intense motion. This fusion method takes into account more complex inter-frame information interactions, not just the base frame as the reference frame, but all frames as reference frames to comprehensively consider all inter-frame information, which can retain more effective intra-frame information and inter-frame information, and has a better fusion effect.

[0074] Specifically, the fusion implementation process can also be expressed as:

[0075]

[0076] Among them, A 1,1 °F 1 is the self-similarity feature of the base frame, Fusion of weighted features for inter-frame similarity will mitigate the adverse effects of misalignment due to intense motion. Further considering the actual inter-frame information, as well as the algorithm implementation complexity and resource occupancy, only the base frame can be used as a reference frame, which has a higher fusion efficiency.

[0077] Step 105: Reconstruct the image according to the fused feature information to obtain a super-resolution image.

[0078] In some embodiments, image reconstruction is performed based on the fused feature information to obtain a super-resolution image, including: reconstructing high-frequency information of the fused feature information based on a self-attention mechanism to obtain a reconstruction result; upsampling the reconstruction result to obtain a super-resolution image. The self-attention mechanism is used to further aggregate the information to perform finer high-frequency detail reconstruction.

[0079] Exemplarily, high-frequency information is reconstructed on the fused feature information based on a self-attention mechanism using at least two layers of stacked encoding-decoding networks to obtain a reconstruction result; that is, high-frequency information can be reconstructed using a stacked two-layer, three-layer, four-layer or more-layer encoding-decoding network.

[0080] Among them, the encoding and decoding network includes a cascaded encoder and decoder, the encoder includes feature extraction based on the self-attention mechanism, and the decoder performs feature reconstruction based on the self-attention mechanism; the output information of the encoder of the previous encoding and decoding network is input into the encoder of the next encoding and decoding network, and the output information of the decoder of the next encoding and decoding network is input into the decoder of the previous encoding and decoding network.

[0081] Exemplarily, the encoder includes three stacked first self-attention modules, and the decoder includes three stacked second self-attention modules; the first self-attention network module includes: a first normalization layer, a first multi-head self-attention network, a second normalization layer, and a first locally enhanced feedforward network layer; the first self-attention network module includes: a third normalization layer, a second multi-head self-attention network, a fourth normalization layer, and a second locally enhanced feedforward network layer.

[0082] In some embodiments, any one of the multi-frame super-resolution reconstruction methods from step 101 to step 105 is implemented by establishing a multi-frame super-resolution reconstruction model.

[0083] It should be noted that the multi-frame super-resolution reconstruction model includes a training phase and a use phase. In the training phase, the method further includes: obtaining a training data set; iteratively executing any one of the steps of the multi-frame super-resolution reconstruction method from step 101 to step 105 based on the training data set until the loss function converges, thereby obtaining a trained multi-frame super-resolution reconstruction model. In the use phase, the multi-frame super-resolution reconstruction model is used to perform super-resolution reconstruction on low-resolution multi-frame images to obtain a super-resolution image.

[0084] Figure 4 The structure of the multi-resolution reconstruction model in the embodiment of the present application is shown in FIG. Figure 1 ,like Figure 4As shown, the multi-frame super-resolution reconstruction model includes: an alignment unit, a feature extraction unit, a joint similarity fusion unit and a reconstruction unit.

[0085] In one embodiment of the present application, an alignment unit is used to align physical jitters between low-resolution multi-frame image sequences by using a homography matrix; pixel displacements in a multi-frame image sequence taken in rapid succession are mainly caused by camera motion and scene changes, and these pixel displacements are generally considered to be supplementary detail information that can be used in reconstruction; before fusion, the multi-frame image sequence needs to be aligned first to avoid artifacts or erroneous super-resolution results caused by information confusion or misalignment;

[0086] Furthermore, the alignment unit is used to align the physical jitter between the first frame and other frames in the low-resolution multi-frame image sequence by calculating the homography matrix between the first frame and other frames in the low-resolution multi-frame image sequence.

[0087] As can be seen from Table 1, using the homography matrix to align the physical jitter between low-resolution multi-frame image sequences can greatly improve the results.

[0088] Table 1 Ablation experiment results of alignment module

[0089] Alignment Fusion Decoding PSNR↑ SSIM↑ LPIPS↓ No alignment FAF(ours) Ours 30.223 0.878 0.125 Optical flow FAF(ours) Ours 30.857 0.889 0.117 Deformable FAF(ours) Ours 30.782 0.891 0.111 Homography FAF(ours) Ours 31.012 0.898 0.102

[0090] In one embodiment of the present application, a feature extraction unit is used to extract preliminary features of an aligned low-resolution multi-frame image sequence through a convolutional neural network, and the extracted features are used to perform further feature fusion in conjunction with a similarity fusion unit.

[0091] In one embodiment of the present application, a joint similarity fusion unit is used to weight the fusion of multi-frame information by utilizing the similarity of inter-frame and intra-frame information of low-resolution multi-frame images; in order to fully explore the potential information, the aligned multi-frame image sequence will be fused. The fused result should be consistent with the base frame while incorporating supplementary information from other frames as much as possible. The joint similarity fusion unit is used to aggregate inter-frame and intra-frame information.

[0092] Furthermore, a joint similarity fusion unit is used to perform weighted fusion according to the similarity differences between low-resolution multi-frame image sequences, focusing on the complementary information between the multiple frames, so as to obtain a weighted fusion result with richer and more comprehensive information.

[0093] In some embodiments, another similarity fusion module (Vanilla Affinity Fusion, VAF) is used to perform weighted fusion on the feature information to focus on pixels similar to the base frame (first frame) in other frames among the multiple frames, which is expressed as:

[0094]

[0095] Among them, F i is the feature information of the i-th frame, A 1,i is the similarity weight between the first frame feature and the i-th frame feature, ° is the Hadamard product; A 1,i It can also be understood as a fusion weight, which is used to weightedly fuse the feature information of multiple frames.

[0096] The federated affinity fusion unit includes: a federated affinity fusion (FAF) module, which is used to implement the similarity weighted fusion of feature information of multiple frames. The FAF module is used to calculate the similarity weight of each pixel between a certain frame and a reference frame, further calculate the fusion weight of each pixel on each frame according to the similarity weight, and perform similarity weighted fusion according to the fusion weight to determine the final result; here, the fusion weight is the difference between the similarity weights of the two frames, where the similarity weight is the dot product result of the feature information of the two frames.

[0097] Specifically, the FAF module is used to calculate the fusion weight between a certain frame in a multi-frame image sequence and the base frame (the first frame), and the implementation process of the joint similarity fusion unit is expressed as:

[0098] A i,j =F i ·F j

[0099] D 1 =A 1,1 ,D j =d(A 1,j ,A 1,1 ), when j≠1

[0100]

[0101] Specifically, the joint similarity fusion unit includes: one or more FAF modules.

[0102] Figure 5 The structure of the multi-resolution reconstruction model in the embodiment of the present application is shown in FIG. Figure 2 ,like Figure 5 As shown, the multi-frame super-resolution reconstruction model includes: a homography matrix alignment unit, a convolutional feature extraction unit, a joint similarity fusion unit and a reconstruction unit.

[0103] A homography matrix alignment unit, used for calculating the homography matrix between the first frame and other frames in the low-resolution multi-frame image sequence, and aligning the physical jitter between the low-resolution multi-frame image sequence by using the homography matrix;

[0104] A convolution feature extraction unit is used to extract preliminary features of the aligned low-resolution multi-frame image sequence through a convolutional neural network (Convolution);

[0105] A joint similarity fusion unit, used for weighted fusion of multi-frame information by utilizing similarity of inter-frame and intra-frame information of low-resolution multi-frame images;

[0106] The reconstruction unit further aggregates and decodes the information using the self-attention mechanism.

[0107] Furthermore, the federated affinity fusion unit may include a federated affinity fusion (FAF) module for implementing weighted similarity fusion of feature information of multiple frames. The FAF module is used to calculate the similarity weight (A) between a certain frame and a reference frame. 1,k ), and further according to the similarity weight (A 1,k ) to obtain the fusion weight of each frame feature information (D k ), according to the fusion weight (D k ) for each frame feature information (F k ) performs similarity weighted fusion to determine the final result; here, the similarity weight (A 1,k ) is the dot product result of a certain frame and the first frame (Dot Product), and the fusion weight (D k ) is the difference between the similarity weight and the reference similarity weight, and the fusion weight (D k ) and F k The Hadamard product operation (Element-Wise Product) results are accumulated to obtain the fused feature information.

[0108] Furthermore, the joint similarity fusion unit can also be an upgraded fusion unit including multiple FAF modules and accumulation modules, which is used to calculate the similarity difference weight between a certain frame and all other frames in the multi-frame image sequence. The implementation process of the joint similarity fusion unit is expressed as follows:

[0109] A i,j =F i ·F j

[0110] D i =A i,i ,D j =d(A i,j ,A i,i ), when j≠i

[0111]

[0112] The upgraded fusion unit follows a fusion method similar to that of a single FAF module and is an extension of a single FAF module. It takes into account more complex joint interactions of inter-frame information. Instead of just taking the base frame as a reference frame, it takes all frames as reference frames and comprehensively considers all inter-frame information. It can retain more effective intra-frame information and inter-frame information, and has a better fusion effect.

[0113] Figure 6 Schematic diagram of the comparison of the fusion process of the FAF module and the VAF module in the embodiment of the present application, Figure 7 Schematic diagram comparing the fusion results of the FAF module and the VAF module in the embodiment of the present application.

[0114] See also Figure 6 and Figure 7 ,Compared to the FAF module, the VAF module retains the ,information similar to the base frame, e.g. Figure 6 Pixel-A in the image, but cannot preserve important details that only appear in other frames, such as Figure 6 The FAF module fuses the pixels in the image by focusing on the differences between similarities, while retaining the pixels similar to the base frame (e.g. Figure 6 A in the image, while preserving details from other frames (e.g. Figure 6 Pixels in B) More supplementary details.

[0115] Therefore, the FAF module is able to pay attention to complementary details that are not present in the base frame (e.g. Figure 6 The module can correct the fusion information, reduce the misalignment of violent motion areas in multiple frames, avoid the negative impact of reconstructed areas with high similarity (such as planar areas), and promote more sub-pixels to be fused, see Figure 7 In addition, the upgraded fusion unit not only takes the basic frame as a reference, but also calculates the similarity difference between a certain frame and all other frames in the multi-frame to obtain the fusion weight.

[0116] This embodiment combines the similarity of multiple frames to guide the weighted fusion of multiple frame information, and finally obtains a super-resolution result with richer details, such as Figure 6 and Figure 7 As shown, the super-resolution result of this embodiment is clearer than that of other algorithms, the detail texture is more accurate, and the visual effect is better.

[0117] It can be seen from Table 2 that by weighting and fusing multi-frame information based on the similarity differences between low-resolution multi-frame image sequences, the super-resolution result can be greatly improved. Among them, FAF* represents the upgrade fusion unit.

[0118] Table 2 Ablation experiment results of fusion module

[0119] Alignment Fusion Decoding PSNR↑ SSIM↑ LPIPS↓ Homography VFA / TSA Ours 30.724 0.896 0.107 Homography FAF(ours) Ours 31.012 0.898 0.102 Homography FAF*(ours) Ours 31.197 0.899 0.101

[0120] In one embodiment of the present application, a reconstruction unit is used to further aggregate and decode information by utilizing a self-attention mechanism; in order to aggregate global information for finer high-frequency detail reconstruction, a multi-frame information decoding and reconstruction unit utilizes a self-attention mechanism to model long-distance pixel relationships; specifically, the multi-frame information decoding and reconstruction unit uses a multi-frame information decoding unit to explicitly model intra-channel interdependencies.

[0121] Furthermore, the reconstruction unit, which is composed of at least two stacked layers of encoding-decoding networks (Encode-Decode) and a pixel reorganization upsampling layer (PixelShuffle), is used to aggregate global information for finer high-frequency detail reconstruction;

[0122] The encoding-decoding network consists of an encoder and a decoder. The encoder extracts features based on the self-attention mechanism, and the decoder reconstructs features based on the self-attention mechanism.

[0123] The encoder includes three stacked first self-attention modules, and the decoder includes three stacked second self-attention modules; the first self-attention module and the second self-attention module have the same composition structure, but different network parameters.

[0124] The first self-attention module of the encoding layer includes: the first normalization layer (LayerNorm), the first multi-head self-attention network (Milti-head SA), the second normalization layer (LayerNorm) and the first locally enhanced feedforward network layer (Locally-enhanced Feed-Forward, LEFF), which are used to further perform weighted aggregation of global information.

[0125] The second self-attention network module of the decoder includes: a third normalization layer (LayerNorm), a second multi-head self-attention network (Milti-head SA), a fourth normalization layer (LayerNorm) and a second locally enhanced feedforward network layer (LEFF).

[0126] The pixel reorganization upsampling layer ((PixelShuffle)) is used to upsample the reconstruction result to the resolution size of the high-resolution image.

[0127] It can be seen from Table 3 that compared with further aggregating information by using the self-attention mechanism, the super-resolution results can be effectively improved.

[0128] Table 3 Ablation experiment results of decoding module

[0129] Alignment Fusion Decoding PSNR↑ SSIM↑ LPIPS↓ Homography FAF(ours) BSRT 30.890 0.895 0.106 Homography FAF(ours) Ours 31.012 0.898 0.102

[0130] It can be seen from Tables 4, 5 and 6 that, compared with different super-resolution models, the multi-frame image super-resolution model based on joint multi-frame similarity guidance (referred to as "FBAnet") provided in the embodiment of the present application has achieved significant improvement in the reconstruction quality on the BurstSR dataset and the RGB domain and RAW domain versions of the RealBSR dataset.

[0131] Table 4 Experimental results of different super-resolution models on the BurstSR dataset

[0132]

[0133] Table 5 Experimental results of different super-resolution models on the RealBSR-RGB dataset

[0134]

[0135] Table 6 Experimental results of different super-resolution models on the RealBSR-RAW dataset

[0136] Method PSNR↑ SSIM↑ LPIPS↓ PSNR-Linear↑ DBSR 20.906 0.635 0.134 30.484 MFIR 21.562 0.638 0.131 30.979 BSRT 22.579 0.662 0.103 30.826 BIPNet 22.896 0.641 0.144 31.311 FBAnet(ours) 23.423 0.677 0.125 32.256

[0137] The multi-frame super-resolution reconstruction model (FBAnet) provided in the embodiment of the present application focuses on the similarity difference between multi-frame image sequences and weightedly fuses the feature information of multiple frames, such as Figure 8 and Fig. 9 As shown in the figure, a super-resolution result with richer details is finally obtained, which effectively improves the image quality.

[0138] To implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application also provides a multi-frame super-resolution reconstruction device, such as Fig.10 As shown, the multi-frame super-resolution reconstruction device 100 includes:

[0139] Alignment unit 1001, feature extraction unit 1002, joint similarity fusion unit 1003 and reconstruction unit 1004;

[0140] An alignment unit 1001 is used to perform alignment processing on multiple frames of first images to obtain multiple aligned frames of second images;

[0141] A feature extraction unit 1002 is used to extract features from multiple frames of second images to obtain feature information of the multiple frames of second images;

[0142] The joint similarity fusion unit 1003 is used to determine the fusion weight of the feature information based on the similarity weights between the feature information of the multiple frames of the second image; based on the fusion weights of the feature information, perform similarity weighted fusion on the feature information of the multiple frames of the second image to obtain fused feature information;

[0143] The reconstruction unit 1004 is used to reconstruct the image according to the fused feature information to obtain a super-resolution image.

[0144] In some embodiments, the alignment unit 1001 is specifically used to determine the homography matrix between the reference frame and other frames in the multiple frames of the first image; and use the homography matrix to align the other frames with the reference frame to obtain the aligned multiple frames of the second image.

[0145] In some embodiments, the feature extraction unit 1002 is used to extract features from multiple frames of second images through a convolutional neural network to obtain feature information of the multiple frames of second images.

[0146] In some embodiments, the joint similarity fusion unit 1003 includes a first fusion module, and the first fusion module is specifically used to perform the following steps:

[0147] The i-th frame in the multiple frames of the second image is taken as a reference frame, and a reference similarity weight is determined according to feature information of the i-th frame; the similarity weight between feature information of the i-th frame and feature information of the j-th frame is determined according to feature information of the i-th frame and feature information of the j-th frame in the multiple frames of the second image; wherein the value of j is not equal to i; the fusion weight of the feature information is determined according to the reference similarity weight and the similarity weight between feature information of the i-th frame and feature information of the j-th frame.

[0148] In some embodiments, the first fusion unit is specifically used to perform the following steps: perform a dot multiplication operation on the feature information of the i-th frame and the feature information of the i-th frame to obtain a similarity weight between the feature information of the i-th frame and the feature information of the i-th frame as a reference similarity weight; perform a dot multiplication operation on the feature information of the i-th frame and the feature information of the j-th frame in multiple frames of the second image to obtain a similarity weight between the feature information of the i-th frame and the feature information of the j-th frame.

[0149] In some embodiments, the i-th frame includes a preset frame among multiple frames of the second image.

[0150] In some embodiments, the first fusion unit is specifically configured to perform the following steps:

[0151] When the i-th frame is used as a reference frame, similarity weighted fusion is performed based on the fusion weights of all frames and the feature information of all frames to determine the fusion feature information corresponding to the i-th frame as a reference frame.

[0152] In some embodiments, the first fusion unit is specifically used to perform the following steps: perform a Hadamard product operation on the fusion weight of the kth frame and the feature information of the kth frame to obtain a Hadamard product operation result; wherein the value of k is equal to i or not equal to i; and fuse the Hadamard product operation results corresponding to multiple frames of the second image to obtain the corresponding fusion feature information when the i-th frame is used as a reference frame.

[0153] In some embodiments, the i-th frame includes at least two frames preset in the plurality of frames of the second image;

[0154] The joint similarity fusion unit 1003 includes at least two first fusion modules and second fusion modules; wherein the i-th first fusion unit is used to obtain fusion feature information corresponding to the i-th frame when the i-th frame is used as a reference frame;

[0155] The second fusion module is used to fuse at least two sets of fusion feature information corresponding to at least two preset frames respectively used as reference frames to obtain final fusion feature information.

[0156] In some embodiments, the reconstruction unit 1004 is used to reconstruct high-frequency information of the fused feature information based on a self-attention mechanism to obtain a reconstruction result; and upsample the reconstruction result to obtain a super-resolution image.

[0157] In some embodiments, the reconstruction unit 1004 includes: at least two layers of stacked encoding and decoding networks. The encoding and decoding network includes a cascaded encoder and a decoder, the encoder extracts features based on a self-attention mechanism, and the decoder reconstructs features based on a self-attention mechanism;

[0158] The output feature map of the encoder of the previous encoding and decoding network is input to the encoder of the next encoding and decoding network, and the output feature map of the decoder of the next encoding and decoding network is input to the decoder of the previous encoding and decoding network.

[0159] In some embodiments, the encoder includes three stacked first self-attention modules, and the decoder includes three stacked second self-attention modules; the first self-attention network module includes: a first normalization layer, a first multi-head self-attention network, a second normalization layer, and a first locally enhanced feedforward network layer; the second self-attention network module includes: a third normalization layer, a second multi-head self-attention network, a fourth normalization layer, and a second locally enhanced feedforward network layer.

[0160] In practical applications, the above-mentioned device can be an electronic device for implementing multi-frame super-resolution tasks, or a chip applied to an electronic device. In the present application, the device can implement the functions of multiple units by means of software, hardware, or a combination of software and hardware, so that the device can execute the multi-frame super-resolution reconstruction method provided in any of the above embodiments. And the technical effects of each technical solution of the device can refer to the technical effects of the corresponding technical solutions in the multi-frame super-resolution reconstruction method, and this application will not go into details one by one.

[0161] Based on the hardware implementation of each unit in the above multi-frame super-resolution reconstruction device, the embodiment of the present application also provides a multi-frame super-resolution reconstruction device, such as Fig.11 As shown, the multi-frame super-resolution reconstruction device 110 includes: a processor 1101 and a memory 1102 configured to store a computer program that can be run on the processor; wherein the processor 1101 is configured to execute the method steps in the aforementioned embodiment when running the computer program.

[0162] Of course, in practical applications, Fig.11 As shown, the various components in the multi-frame super-resolution reconstruction device 110 are coupled together through a bus system 1103. It can be understood that the bus system 1103 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1103 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are marked as bus system 1103 in the figure.

[0163] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0164] The above-mentioned memory may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor. The memory may be a separate device independent of the processor, or may be integrated in the processor.

[0165] Optionally, the device may further include an input interface, wherein the processor may control the input interface to communicate with other devices or chips, and specifically, may acquire multiple frames of low-resolution images sent by other devices or chips.

[0166] Optionally, the device may further include an output interface, wherein the processor may control the output interface to communicate with other devices or chips, and specifically, may output the super-resolution image to other devices or chips.

[0167] In an exemplary embodiment, the present application also provides a computer-readable storage medium, such as a memory including a computer program, and the computer program can be executed by a processor of a multi-frame super-resolution reconstruction device to complete the steps of the aforementioned method.

[0168] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0169] Optionally, the computer program product can be applied to the multi-frame super-resolution reconstruction device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the multi-frame super-resolution reconstruction device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0170] The embodiment of the present application also provides a computer program.

[0171] Optionally, the computer program can be applied to the multi-frame super-resolution reconstruction device in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the multi-frame super-resolution reconstruction device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0172] It should be understood that the terms used in the present application are only for the purpose of describing specific embodiments, but are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "having", "may have", "include" and "include", or "may include" and "may include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but the presence of additional features is not excluded.

[0173] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0174] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0175] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0176] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0177] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0178] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A multi-frame super-resolution reconstruction method, characterized in that: The method comprises: Performing alignment processing on the multiple frames of first images to obtain the aligned multiple frames of second images; Performing feature extraction on the multiple frames of second images to obtain feature information of the multiple frames of second images; Determining a fusion weight of the feature information based on similarity weights between the feature information of the multiple frames of second images; Based on the fusion weight of the feature information, similarity-weighted fusion is performed on the feature information of the multiple frames of the second image to obtain fused feature information; Image reconstruction is performed according to the fused feature information to obtain a super-resolution image.

2. The method according to claim 1, characterized in that The step of performing alignment processing on the multiple frames of first images to obtain the aligned multiple frames of second images includes: Determine a homography matrix between a reference frame and other frames in the plurality of first image frames; The other frames are aligned with the reference frame using the homography matrix to obtain aligned multiple frames of second images.

3. The method according to claim 1, characterized in that The determining the fusion weight of the feature information based on the similarity weights between the feature information of the multiple frames of second images includes: Taking the i-th frame of the multiple frames of second images as a reference frame, and determining a reference similarity weight according to feature information of the i-th frame; Determining a similarity weight between the feature information of the i-th frame and the feature information of the j-th frame according to the feature information of the i-th frame and the feature information of the j-th frame in the plurality of frames of the second image; wherein the value of j is not equal to i; The fusion weight of the feature information is determined according to the reference similarity weight and the similarity weight between the feature information of the i-th frame and the feature information of the j-th frame.

4. The method according to claim 3, characterized in that The step of determining the reference similarity weight according to the feature information of the i-th frame includes: Performing a dot multiplication operation on the feature information of the i-th frame and the feature information of the i-th frame to obtain a similarity weight between the feature information of the i-th frame and the feature information of the i-th frame as the reference similarity weight; The determining, according to the feature information of the i-th frame and the feature information of the j-th frame in the multiple frames of the second image, the similarity weight of the feature information of the i-th frame and the feature information of the j-th frame comprises: A point multiplication operation is performed on the feature information of the i-th frame and the feature information of the j-th frame in the multiple frames of the second image to obtain a similarity weight between the feature information of the i-th frame and the feature information of the j-th frame.

5. The method according to claim 3, characterized in that: Determining the fusion weight of the feature information according to the reference similarity weight and the similarity weight between the feature information of the i-th frame and the feature information of the j-th frame includes: Using the reference similarity weight as the fusion weight of the feature information of the i-th frame; A difference operation is performed between the similarity weights of the feature information of the i-th frame and the feature information of the j-th frame and the reference similarity weight to obtain a fusion weight of the feature information of the j-th frame.

6. The method according to any one of claims 3 to 5, characterized in that: The i-th frame includes a preset frame among the multiple frames of second images.

7. The method according to any one of claims 3 to 5, characterized in that: The similarity-weighted fusion of the feature information of the plurality of frames of the second image based on the fusion weight of the feature information to obtain fused feature information includes: When the i-th frame is used as a reference frame, similarity weighted fusion is performed according to the fusion weights of all frames and the feature information of all frames to determine the fusion feature information corresponding to the i-th frame when it is used as a reference frame.

8. The method according to claim 7, characterized in that The performing similarity weighted fusion on the feature information of all frames according to the fusion weights of all frames to determine the fusion feature information corresponding to the i-th frame as the reference frame includes: Performing a Hadamard product operation on the fusion weight of the k-th frame and the feature information of the k-th frame to obtain a Hadamard product operation result; wherein the value of k is equal to i or not equal to i; The Hadamard product operation results corresponding to the multiple frames of the second image are fused to obtain the fusion feature information corresponding to the i-th frame as the reference frame.

9. The method according to claim 7, characterized in that: The i-th frame includes at least two frames preset in the multiple frames of the second image; The method further comprises: performing similarity-weighted fusion on the feature information of the plurality of frames of the second image based on the fusion weight of the feature information to obtain fused feature information; At least two sets of fused feature information corresponding to the at least two preset frames are respectively used as reference frames to obtain final fused feature information.

10. The method according to claim 1, characterized in that The step of reconstructing an image according to the fused feature information to obtain a super-resolution image comprises: Reconstructing high-frequency information of the fused feature information based on a self-attention mechanism to obtain a reconstruction result; The reconstruction result is subjected to pixel reorganization and upsampling to obtain the super-resolution image.

11. The method according to claim 10, characterized in that The high-frequency information reconstruction of the fused feature information based on the self-attention mechanism to obtain a reconstruction result includes: Reconstructing high-frequency information of the fused feature information using at least two layers of stacked encoding and decoding networks based on a self-attention mechanism to obtain a reconstruction result; The encoding and decoding network includes a cascaded encoder and decoder, the encoder performs feature extraction based on a self-attention mechanism, and the decoder performs feature reconstruction based on a self-attention mechanism; The output information of the encoder of the previous encoding and decoding network is input to the encoder of the next encoding and decoding network, and the output information of the decoder of the next encoding and decoding network is input to the decoder of the previous encoding and decoding network.

12. The method according to claim 11, characterized in that The encoder includes three stacked first self-attention modules, and the decoder includes three stacked second self-attention modules; The first self-attention network module includes: a first normalization layer, a first multi-head self-attention network, a second normalization layer, and a first locally enhanced feedforward network layer; The second self-attention network module includes: a third normalization layer, a second multi-head self-attention network, a fourth normalization layer and a second locally enhanced feedforward network layer.

13. A multi-frame super-resolution reconstruction device, characterized in that: The device comprises: an alignment unit, a feature extraction unit, a joint similarity fusion unit and a reconstruction unit; The alignment unit is used to perform alignment processing on the multiple frames of first images to obtain the aligned multiple frames of second images; The feature extraction unit is used to extract features from the multiple frames of second images to obtain feature information of the multiple frames of second images; The joint similarity fusion unit is used to determine the fusion weight of the feature information based on the similarity weights between the feature information of the multiple frames of the second image; based on the fusion weights of the feature information, perform similarity weighted fusion on the feature information of the multiple frames of the second image to obtain fused feature information; The reconstruction unit is used to perform image reconstruction according to the fused feature information to obtain a super-resolution image.

14. A multi-frame super-resolution reconstruction device, characterized in that: The device comprises: a processor and a memory configured to store a computer program capable of being executed on the processor, Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 12 when running the computer program.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images