Continuous shooting image super-resolution method based on discrete wavelet transform fusion

Abstract

The invention relates to the technical field of super-resolution reconstruction, and discloses a continuous shooting image super-division method based on discrete wavelet transform fusion, which comprises the following steps of: extracting features of multiple frames of low-resolution pictures, and aligning the pictures by using deformable convolution to obtain aligned low-resolution pictures; decomposing the feature map of the low-resolution picture into low-frequency components and high-frequency components in different directions to obtain preliminary fusion features; separating the preliminary fusion feature channel into a semantic branch and a spatial branch, and obtaining a fused feature map by using complementarily enhanced semantic features and spatial features; and carrying out channel expansion on the fused feature map by using a pixel shuffling algorithm, and carrying out spatial rearrangement on the features after channel expansion to realize efficient and rapid up-sampling. According to the method, the problems of difficulty in alignment of multi-target complex movement and artifacts, detail distortion, blurring and the like of super-resolution of different-scale targets in a traditional method are solved, and the multi-image super-resolution imaging effect is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of super-resolution reconstruction, in particular to a burst image super-resolution method based on discrete wavelet transform fusion. BACKGROUND

[0002] The burst image super-resolution method is derived from the contradiction between the actual demand for high-resolution images and the limitations of single-image super-resolution technology. The goal of super-resolution is to recover high-resolution images from low-resolution images, making up for the lack of hardware in resolution. Single-image super-resolution (SISR) only relies on a single LR image for reconstruction, but the information contained in a single image is limited, and it is an ill-posed problem. The single-image super-resolution reconstruction result will have unnatural edges and cannot meet the actual scene demand for high-fidelity details. To break through the limitations of single-image information, multi-image super-resolution technology has developed rapidly. It uses the camera to quickly take multiple low-resolution images of the same scene. Due to slight shaking when holding the camera and movement of objects when taking pictures, different images capture different details of the scene, so the low-resolution images taken in bursts usually contain a variety of information. By fusing multiple information-rich images, more realistic high-frequency details can be recovered, while noise and artifacts can be suppressed.

[0003] In the burst image super-resolution technology, the first low-resolution image is called the base frame, and the other low-resolution images are called other frames, which provide sub-pixel information for reconstructing high-quality high-resolution images. The reconstruction process of burst image super-resolution mainly includes three parts: alignment, fusion, and upsampling. Among them, fusion is the aggregation of information in low-resolution images, and the sub-pixel information matching the content of the base frame is extracted from other frames to provide means for the reconstruction of high-resolution images. Most current work extracts features in the spatial domain, and then fuses the remaining frames with the base frame in the spatial domain or Fourier domain. With the development of deep learning, it provides a lot of technical support for the development of burst super-resolution. End-to-end models based on CNN and Transformer can automatically learn the motion correlation and feature complementarity between images, replacing the manually designed alignment and fusion modules in traditional super-resolution technology, and significantly improving the reconstruction accuracy and robustness.

[0004] For example, Chinese patent application CN202411681817.X discloses a multi-image super-resolution imaging method guided by a frequency-domain low-pass filter. This method transforms low-resolution source images from the spatial domain to the frequency domain to obtain a spectrogram; performs low-pass filtering on the spectrogram to extract structural and detail information from the source images, generating an initial weight map; locates boundary regions and fuses multiple source images; estimates optical flow based on global motion multi-resolution, capturing the direction and displacement information between adjacent pixels in the fused image, and determines the weights of boundary regions based on this information; finally, combines the initial weight map and the weights of the boundary regions to perform weighted fusion of multiple source images, generating a high-resolution image. This method generates detail features from low-resolution images under different focusing conditions, producing a high-resolution image that is clear in all regions, thereby improving the accuracy and computational efficiency of multi-image super-resolution.

[0005] However, traditional multi-image super-resolution imaging methods have the following defects or shortcomings:

[0006] 1) The design of traditional optical flow alignment methods relies on several simplifying assumptions, which are broken in non-rigid motion and complex motion scenarios, leading to the generation of initial errors and the problem of error accumulation;

[0007] 2) Information loss problem in the fusion of low-resolution images in the spatial and Fourier domains. The fusion of features in the spatial domain of low-resolution images leads to the loss of "structural details" and "momentation" blurring in local areas. The fusion in the Fourier domain suffers from the problem of spatial structure restoration failure due to phase information distortion.

[0008] 3) Lack of attention to spatial location and channel dimension in the feature fusion stage. Target information and complex texture features at different scales in burst images may be scattered across different channels and spatial locations, posing challenges to fusion and reconstruction.

[0009] No effective solutions have yet been proposed to address the problems in the relevant technologies. Summary of the Invention

[0010] To address the problems in related technologies, this invention proposes a super-resolution method for continuous image capture based on discrete wavelet transform fusion, in order to overcome the aforementioned technical problems existing in existing related technologies.

[0011] This invention uses discrete wavelet transform to analyze low-frequency components and high-frequency components with different directions in low-resolution images. It employs deformable convolution to align features from multiple frames and uses bidirectional path-guided fusion combining spatial attention and second-order channel attention mechanisms to achieve a fusion of image detail features and hierarchical structure.

[0012] Therefore, the specific technical solution adopted by the present invention is as follows:

[0013] A super-resolution method for continuous image capture based on discrete wavelet transform fusion includes the following steps:

[0014] S1. Extract features from multiple low-resolution images based on the encoder, and use deformable convolution to calculate the offset between the multiple low-resolution images and the base frame. Align the image features according to the offset to obtain the feature map of the low-resolution image.

[0015] S2. Using discrete wavelet transform technology, the feature map of the low-resolution image is decomposed into low-frequency components and high-frequency components with different directions, and the preliminary fused features are obtained by fusing wavelet features and residual features.

[0016] S3. Based on the bidirectional guidance mechanism, the initial fused feature channels are separated into semantic branches and spatial branches, and spatial attention and second-order channel attention are used to guide the semantic features and spatial features to complement each other and enhance them, so as to obtain the fused feature map.

[0017] S4. The pixel shuffling algorithm is used to expand the channels of the fused feature map and then the expanded features are spatially rearranged to convert the channel information into spatial information, thereby achieving efficient and fast upsampling.

[0018] Furthermore, based on the encoder, features of multiple low-resolution images are extracted, and deformable convolution is used to calculate the offset between the multiple low-resolution images and the base frame. The image features are then aligned according to the offset to obtain the feature map of the low-resolution image, including the following steps:

[0019] S11. Use an encoder to extract features from multiple frames of low-resolution images to convert pixel-level information into feature-level information;

[0020] S12. Based on the feature maps of the base frame and the current frame, construct a cascaded multi-scale deformable convolution alignment module, and calculate the offset required for deformable convolution of each current frame and the base frame at different scale levels.

[0021] S13. Based on the offset, sample the feature map of the current level, and use the deformable convolution alignment module in combination with the preset concatenation order to calculate the offset estimation and feature sampling on the feature map of each scale level by level.

[0022] S14. Local features are obtained by utilizing the receptive field of the low-level feature map, and global features are captured by utilizing the receptive field of the high-level feature map. This achieves motion compensation from coarse to fine, resulting in a feature map of the low-resolution image after feature alignment.

[0023] Furthermore, the formula for calculating the offset is:

[0024] ΔP k (p0)=Conv([fb f c ]);

[0025] The expression for the feature map of the low-resolution image after feature alignment is:

[0026]

[0027] In the formula, ΔP k (p0) represents the offset between the predicted current frame and the base frame, k represents the sampling point, p0 represents the center position of the feature map, Conv represents the convolutional layer, and f b f represents the features of the base frame. c Y represents the features of the current frame. o (p0) represents the feature map of the low-resolution image after feature alignment, K represents the position of the low-resolution image within the convolution kernel, and w k The weights of the convolution kernel are represented by pk, pk represents the sampled convolutional layer, and f represents the current convolutional layer.

[0028] Furthermore, using discrete wavelet transform technology, the feature map of the low-resolution image is decomposed into low-frequency components and high-frequency components with different directions. The preliminary fused features are obtained by fusing wavelet features with residual features, including the following steps:

[0029] S21. Using discrete wavelets, the feature map of a low-resolution image is decomposed into four sub-bands: a low-frequency component containing low-frequency elements reflecting the image and a high-frequency component containing image edges and textures, to obtain the decomposed multi-component features.

[0030] S22. Using the wavelet feature concatenation mechanism, the decomposed multi-component features are concatenated into a feature tensor, and then convolution and activation functions are applied to obtain high-dimensional features.

[0031] S23. Construct residual paths and use parallel convolutional paths to preserve the original feature information in order to mitigate feature loss caused by wavelet transform.

[0032] S24. The residuals of the wavelet features and residual features are summed and used as the output of the discrete wavelet transform fusion to obtain the preliminary fused features.

[0033] Furthermore, the four sub-bands include a low-frequency sub-band, a horizontal high-frequency sub-band, a vertical high-frequency sub-band, and a diagonal high-frequency sub-band;

[0034] Among them, the low-frequency subband is used to reflect the low-frequency components of the image and describe the overall contour and smooth areas; the horizontal high-frequency subband is used to capture horizontal edges and details; the vertical high-frequency subband is used to extract vertical edge information; and the diagonal high-frequency subband is used to reflect edge features and texture details in the diagonal direction.

[0035] Furthermore, the expression for the low-frequency sub-band is:

[0036]

[0037] The expression for the horizontal high-frequency subband is:

[0038]

[0039] The expression for the vertical high-frequency subband is:

[0040]

[0041] The expression for the diagonal high-frequency subband is:

[0042]

[0043] In the formula, Y LL (i, j), Y LH (i, j), Y HL (i, j), Y HH (i, j) represent the low-frequency sub-band, horizontal high-frequency sub-band, vertical high-frequency sub-band, and diagonal high-frequency sub-band, respectively, and Y(i, j) represents the feature value of the aligned input image feature at index (i, j).

[0044] Furthermore, based on a bidirectional guidance mechanism, the initially fused feature channels are separated into semantic and spatial branches, and spatial attention and second-order channel attention are used respectively to guide the semantic and spatial features to perform complementary enhancements, resulting in a fused feature map including the following steps:

[0045] S31. Separate the initial fused feature channels into semantic branches and spatial branches to process high-level semantic information and low-level spatial details respectively;

[0046] S32. Use spatial attention to guide semantic features to focus on key spatial regions, and use second-order channel attention to guide spatial features to focus on key semantic channels, so as to achieve complementary enhancement of the two features.

[0047] S33. Based on the adaptive feature allocation method, the channel ratio of the main branch and the sub-branch is dynamically allocated to ensure that spatial details are not compressed while retaining sufficient semantic information, so as to obtain the fused feature map.

[0048] Furthermore, spatial attention is used to guide semantic features to focus on key spatial regions, and second-order channel attention is used to guide spatial features to focus on key semantic channels, including:

[0049] Spatial attention is used to enhance the semantic complementarity of spatial features, thereby guiding semantic features to focus on key spatial regions;

[0050] We utilize second-order channel attention to enhance the semantic guidance of feature maps in the space, thereby guiding spatial features to focus on key semantic channels.

[0051] Furthermore, second-order channel attention captures higher-order correlations between channels through second-order statistics to enhance feature representation capabilities, and achieves global interaction between channels through eigenvalue decomposition of the covariance matrix to overcome the limitations of traditional first-order statistics.

[0052] Furthermore, the pixel shuffling algorithm is used to perform channel expansion on the fused feature map, and the channel-expanded features are spatially rearranged to convert channel information into spatial information, achieving efficient and fast upsampling. This includes the following steps:

[0053] S41. The number of channels of the fused feature map is expanded to a preset multiple of the super-resolution multiple using convolutional layers, and the spatial information of the low-resolution image is encoded into more channels to store information for subsequent spatial dimension expansion.

[0054] S42. Using a shuffling operation, the channel index in the expanded high-channel feature map is decomposed into the channels of the output image and the index for spatial rearrangement. Then, combined with spatial mapping, the corresponding position of the input feature map is mapped to the spatial position of the high-resolution image.

[0055] Compared with existing technologies, this invention provides a super-resolution method for continuous image capture based on discrete wavelet transform fusion, which has the following beneficial effects:

[0056] (1) This invention utilizes direction-sensitive discrete wavelet transform to decouple frequency components in images and uses bidirectional guided fusion of spatial attention and channel attention to overcome the artifacts of super-resolution of targets at different scales, as well as the problems of local detail distortion and blurring that exist in traditional methods, thereby enhancing the effect of multi-image super-resolution imaging.

[0057] (2) This invention utilizes deformable convolution to perform end-to-end learning of the small displacements of the base frame and the current frame. The cascaded deformable convolution structure processes features at different scales simultaneously, processing global motion at the low-dimensional level and refining local details at the high-dimensional level, thereby aligning the features of continuous shooting images. This can effectively alleviate the error propagation problem of the optical flow method and has better robustness to complex scenes with noise and low light conditions.

[0058] (3) This invention utilizes the advantages of the layered processing and direction sensitivity of discrete wavelet transform to decompose low-resolution images into low-frequency components and high-frequency components with different directions. Then, each component is processed in parallel and subsequently fused by convolution. This can effectively restore the low-frequency and high-frequency details required for reconstructing high-resolution images and accurately capture the local details and non-stationary signals of the image.

[0059] (4) This invention can not only dynamically adjust the feature channel weights at different times using channel attention, but also dynamically locate the spatial position of objects in each frame using spatial attention. The two work together to adapt to the dynamic changes of the scene, enhance the complementarity and comprehensiveness of multi-frame feature fusion, and ensure that the details of objects of different scales in complex scenes are coherent and clear in the reconstructed high-resolution image. Attached Figure Description

[0060] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0061] Figure 1 This is a flowchart of a continuous image super-resolution method based on discrete wavelet transform fusion according to an embodiment of the present invention. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] According to an embodiment of the present invention, a super-resolution method for continuous image capture based on discrete wavelet transform fusion is provided.

[0064] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figure 1 As shown, according to an embodiment of the present invention, a super-resolution method for continuous image capture based on discrete wavelet transform fusion is provided, comprising the following steps:

[0065] S1. Extract features from multiple low-resolution images based on the encoder, and use deformable convolution to calculate the offset between the multiple low-resolution images and the base frame. Align the image features according to the offset to obtain the feature map of the low-resolution image.

[0066] In this embodiment, cascaded deformable convolutions are used to perform alignment operations between the base frame and the current frame, eliminating large inter-frame displacements and accurately estimating small subpixel-level displacements, providing necessary information for detail recovery during the reconstruction process.

[0067] The process of extracting features from multiple low-resolution images based on an encoder, calculating the offset between the multiple low-resolution images and the base frame using deformable convolution, and aligning the image features according to the offset to obtain the feature map of the low-resolution image includes the following steps:

[0068] S11. Use an encoder to extract features from multiple frames of low-resolution images to convert pixel-level information into feature-level information;

[0069] Specifically, the formula for encoding the input image is:

[0070] f i =encoder(p i )

[0071] encoder={ResBlock1, ResBlock2....ResBlock n}

[0072] In the formula, f i p represents the encoded output. i This represents each low-resolution image frame, where the encoder is an n-cascaded residual module ResBlock.

[0073] S12. Based on the feature maps of the base frame and the current frame, construct a cascaded multi-scale deformable convolution alignment module, and calculate the offset required for deformable convolution of each current frame and the base frame at different scale levels.

[0074] S13. Based on the offset, sample the feature map of the current level, and use the deformable convolution alignment module in combination with the preset concatenation order to calculate the offset estimation and feature sampling on the feature map of each scale level by level.

[0075] S14. Utilize the smaller receptive field of low-level feature maps to focus on acquiring local features, and utilize the larger receptive field of high-level feature maps to capture global features, thereby achieving motion compensation from coarse to fine and obtaining feature maps of low-resolution images after feature alignment.

[0076] Specifically, the formula for calculating the offset is:

[0077] ΔP k (p0)=Conv([f b f c ]);

[0078] The expression for the feature map of the low-resolution image after feature alignment is:

[0079]

[0080] In the formula, ΔP k (p0) represents the offset between the predicted current frame and the base frame, k represents the sampling point, p0 represents the center position of the feature map (for each output position, the offset of K sampling points needs to be predicted, and each sampling point has offsets in both the x and y directions), Conv represents the convolutional layer, and fb f represents the features of the base frame. c Y represents the features of the current frame. o (p0) represents the feature map of the low-resolution image after feature alignment, K represents the position of the low-resolution image within the convolution kernel, and w k p represents the weights of the convolution kernel. k This represents the relative coordinates within the convolution kernel, and f represents the current convolutional layer. This sampling method is applicable to feature alignment at different scale levels.

[0081] S2. Using discrete wavelet transform technology, the feature map of the low-resolution image is decomposed into low-frequency components and high-frequency components with different directions, and the preliminary fused features are obtained by fusing wavelet features and residual features.

[0082] The process of using discrete wavelet transform to decompose the feature map of a low-resolution image into low-frequency components and high-frequency components with different directions, and then fusing wavelet features with residual features to obtain preliminary fused features, includes the following steps:

[0083] S21. Using discrete wavelets (discrete wavelet transform with Haar wavelets as the basis function), the feature map of the low-resolution image is decomposed into four sub-bands containing low-frequency components reflecting the low-frequency components of the image and high-frequency components containing the image edges and textures, thus obtaining the decomposed multi-component features.

[0084] Specifically, as a simple and commonly used wavelet basis function, the Haar wavelet exhibits significant temporal locality and is highly sensitive to local abrupt changes and edge details in the signal. Therefore, it can capture the differences between image pixels, thereby extracting edge features and texture information. Thus, the Haar wavelet is chosen as the fundamental function for image decomposition. The input image is divided into four sub-bands using Discrete Wavelet Transform (DWT): the low-frequency (LL) sub-band reflects the low-frequency components of the image, describing the overall contour and smooth regions; the horizontal high-frequency (LH) sub-band captures horizontal edges and details; the vertical high-frequency (HL) sub-band extracts vertical edge information; and the diagonal high-frequency (HH) sub-band reflects edge features and texture details in the diagonal direction. The formula for the two-dimensional image discrete wavelet transform using the Haar wavelet as the basis function is as follows:

[0085]

[0086] In the formula, Y LL (i, j), Y LH (i, j), Y HL (i, j), Y HH(i, j) represent the low-frequency subband (i.e., the low-frequency components of the image), the horizontal high-frequency subband (i.e., horizontal edges and details), the vertical high-frequency subband (i.e., edge features and texture details of vertical edge information), and the diagonal high-frequency subband (i.e., edge features and texture details in the diagonal direction), respectively. Y(i, j) represents the feature value of the aligned input image features at index (i, j).

[0087] S22. Using the wavelet feature concatenation mechanism, the decomposed multi-component features are concatenated into a feature tensor, and then convolution and activation functions are applied to obtain high-dimensional features.

[0088] S23. Construct residual paths and use parallel convolutional paths to preserve the original feature information in order to mitigate feature loss caused by wavelet transform.

[0089] S24. The wavelet features and residual features are summed and used as the output of the discrete wavelet transform fusion to obtain the preliminary fused features;

[0090] In this embodiment, the expression for the features after discrete wavelet transform decomposition using Haar wavelets as the basis function is as follows:

[0091]

[0092] Discrete wavelet transform of a two-dimensional image can be performed using a separable filter. The core idea is to apply the two-dimensional wavelet filter sequentially to each row and column of the image data, thereby decomposing the two-dimensional wavelet transform of the image into two independent one-dimensional wavelet transforms.

[0093] The expression for obtaining the high-dimensional features is:

[0094]

[0095] The expression for the feature information of the residual path is:

[0096] I d =downsample(Conv(I));

[0097] The expression for merging wavelet feature paths and residual paths is:

[0098] Z = Y w +I d ;

[0099] In the formula, * represents the convolution operation, and ↓2 represents downsampling with a sampling factor of 2, using a low-pass filter P. LL and three high-pass filters (vertical direction filter P) LH Horizontal direction filter P HL and diagonal direction filter P HHThe image is transformed by domain transformation to generate a low-frequency sub-band image Y with four sub-bands. LL and three high-frequency subband images Y LH Y HL Y HH Y represents the feature map after wavelet decomposition. w This indicates that the data has undergone convolution and activation processing. [*] indicates a concatenation operation along the channel dimension. Conv is a convolution operation used to adjust channels and size. ReLU is the activation function. d This represents the downsampled residual feature. Downsample is a downsampling operation used to reduce the width and height of the feature map. Z represents the dual-path fusion feature, and + is the fusion operation of wavelet features and residual path features.

[0100] S3. Based on the bidirectional guidance mechanism, the initial fused feature channels are separated into semantic branches and spatial branches, and spatial attention and second-order channel attention are used to guide the semantic features and spatial features to complement each other and enhance them, so as to obtain the fused feature map.

[0101] The bidirectional guidance mechanism involves separating the initial fused feature channels into semantic and spatial branches, and then using spatial attention and second-order channel attention to guide the semantic and spatial features to perform complementary enhancements, resulting in a fused feature map. This process includes the following steps:

[0102] S31. Separate the initial fused feature channels into semantic branches and spatial branches to process high-level semantic information and low-level spatial details respectively;

[0103] Specifically, the expression for separating the wavelet fusion feature channels into semantic and spatial branches is as follows:

[0104] (Z1, Z2) = Split(Z in );

[0105] In the formula, Z1 and Z2 are the inputs Z in The two features separated in the channel dimension have dimensions ac×H×W and (1-a)c×H×W, respectively, where a represents the channel separation ratio, c represents the number of channels, H represents the feature map height, and W represents the feature map width.

[0106] S32. Use spatial attention to guide semantic features to focus on key spatial regions, and use second-order channel attention to guide spatial features to focus on key semantic channels, so as to achieve complementary enhancement of the two features.

[0107] Specifically, the method of using spatial attention to guide semantic features to focus on key spatial regions, and using second-order channel attention to guide spatial features to focus on key semantic channels includes:

[0108] Spatial attention is used to enhance the semantic complementarity of spatial features, thereby guiding semantic features to focus on key spatial regions;

[0109] We utilize second-order channel attention to enhance the semantic guidance of feature maps in the space, thereby guiding spatial features to focus on key semantic channels.

[0110] The second-order channel attention captures higher-order correlations between channels through second-order statistics to enhance feature representation capabilities. It achieves global interaction between channels through eigenvalue decomposition of the covariance matrix to overcome the limitations of traditional first-order statistics.

[0111] S33. Based on the adaptive feature allocation method, the channel ratio of the main branch and the sub-branch is dynamically allocated to ensure that spatial details are not compressed while retaining sufficient semantic information, so as to obtain the fused feature map.

[0112] In this embodiment, the expressions for the spatial attention mechanism of the spatial region and the second-order channel attention mechanism of the semantic branch are as follows:

[0113] W s =σ(Conv 7×7 ([AvgPool(Z1);MaxPool(Z1)]));

[0114] In the formula, W s This is the weight matrix for the spatial attention mechanism, which can be applied to the feature map later. σ is the Sigmoid activation function, AvgPool is the global average pooling operation, MaxPool is the global max pooling operation, and Conv... 7×7 [*] is a convolution operation with a 7x7 kernel, and [*] is a feature concatenation operation;

[0115] Implementation of the second-order channel attention mechanism: First, the input features are reshaped into the input feature map Z2∈R. B×C×H×W Remodeling into X∈R B×C×N In the formula, B represents the batch size, C represents the number of channels, X represents the reshaped feature map, and N represents the height-width product. Next, the covariance matrix ∑ between channels is calculated using the following formula:

[0116]

[0117] In the formula, Z2 ～ These are the reshaped features, where N is the product of height and width, and Z is the height-width product. 2～T This is a transpose matrix, where T represents the matrix transpose, I is an N×N identity matrix, and E is a matrix of all ones. This step eliminates the influence of the mean through a centering operation and extracts the second-order correlation between channels;

[0118] Then, eigenvalue decomposition and covariance normalization are performed on the covariance matrix ∑:

[0119]

[0120] Where U is an orthogonal eigenvector matrix, and Λ is a diagonal matrix of eigenvalues ​​arranged in non-increasing order, represented as Lambda = diag(1ambda_1, lambda_2, ..., lambdaC). This is the result of covariance normalization, which is achieved by adjusting the power α of the eigenvalues ​​(usually α = 0.5). This step enhances the discriminative power of the features through nonlinear scaling.

[0121] Finally, attention weights are generated and the original feature map is multiplied by the attention weights. The feature vector u corresponding to the largest eigenvalue is extracted. max Channel weights are generated using SoftMax normalization, reflecting the importance of each channel. The original feature map Z2 is then multiplied by the attention weights to obtain the enhanced feature map.

[0122]

[0123] In the formula, 'a' represents 'a', and SOCA represents second-order channel attention.

[0124] SOCA captures higher-order correlations between channels using second-order statistics, significantly improving the model's feature representation capabilities in tasks such as image super-resolution. Its core advantage lies in achieving global interaction between channels through eigenvalue decomposition of the covariance matrix, overcoming the limitations of traditional first-order statistics.

[0125] The features processed in parallel are fused to obtain the final output result L, L = Z1 + Z2.

[0126] S4. The pixel shuffling algorithm is used to expand the channels of the fused feature map and then the expanded features are spatially rearranged to convert the channel information into spatial information, thereby achieving efficient and fast upsampling.

[0127] The step of using a pixel shuffling algorithm to perform channel expansion on the fused feature map and spatial rearrangement on the channel-expanded features to convert channel information into spatial information, thereby achieving efficient and fast upsampling, includes the following steps:

[0128] S41. The number of channels of the fused feature map is expanded to a preset multiple (square multiple) of the super-resolution multiple using convolutional layers, and the spatial information of the low-resolution image is encoded into more channels to store information for subsequent spatial dimension expansion.

[0129] Specifically, the formula for expanding the number of channels in a low-resolution image feature map to a square multiple of the super-resolution using convolutional layers is as follows:

[0130] L e=Conv(L);

[0131] I H =Reshape(L e );

[0132] In the formula, L∈R N×C×H×W These are fused features processed by a parallel attention mechanism. It is the result of expanding the channel dimension to a square multiple of the super-resolution factor, I H ∈R N×C×(r×H)×(r×W) It is a reconstructed high-resolution image.

[0133] S42. Using a shuffling operation, the channel index in the expanded high-channel feature map is decomposed into the channels of the output image and the index for spatial rearrangement. Then, combined with spatial mapping, the corresponding position of the input feature map is mapped to the spatial position of the high-resolution image.

[0134] In this embodiment, the alignment method based on deformable convolution allows each sampling point to adjust its position independently, forming a dense sampling pattern with multiple directions and scales. This characteristic makes it perform better when handling non-rigid motion and large displacement scenarios. The end-to-end advantage of deformable convolution can alleviate the error propagation problem of optical flow methods and has better robustness to complex scenarios with noise and low light conditions.

[0135] In the field of super-resolution of burst images, Discrete Wavelet Transform (DWT) has significant advantages over Fourier Transform (FWT) in feature fusion. DWT decomposes the image into sub-bands of different scales, enabling localized analysis in the frequency domain and accurately capturing local details. This characteristic makes it superior in processing local differences in burst images caused by camera shake or minor object movements. In contrast, Fourier Transform, based on global sine wave basis functions, lacks the ability to locate local information and struggles to effectively distinguish frequency component changes in local regions across different frames, easily leading to detail loss or blurring during feature fusion. The directional sensitivity of DWT also allows for the analysis of low-frequency information and high-frequency information with different directions at different scales, enabling more comprehensive extraction of edge and texture information in different directions. Fourier Transform, however, only provides a global frequency distribution and cannot achieve this multi-directional feature separation. Finally, DWT's basis functions can be flexibly selected, adaptively optimizing feature extraction based on the characteristics of burst images, while Fourier Transform's basis functions are fixed, lacking this flexibility.

[0136] In the field of super-resolution of burst images, combining spatial attention and second-order channel attention mechanisms can achieve multi-level optimization in feature processing. Spatial attention focuses on key regions at different locations in the image, enhancing the expression of local details, and is particularly suitable for handling spatial information differences caused by motion or misalignment in burst images. Second-order channel attention, on the other hand, captures the second-order statistical correlation between channels by calculating the covariance matrix of features. Compared to traditional first-order statistics, it can more deeply explore the complex dependencies between channels, improving the discriminative power and robustness of features. The combination of the two mechanisms allows spatial attention to provide spatial prior guidance for channel attention, helping SOCA more accurately locate the channels that need enhancement; while SOCA optimizes feature distribution through second-order interactions between channels, further providing a more discriminative feature foundation for spatial attention, forming a complementary and synergistic effect. This dual attention mechanism not only strengthens the capture of details in key regions in the spatial dimension but also optimizes the global correlation of features in the channel dimension. It is particularly suitable for the fusion of multi-frame information in burst images, and can more effectively utilize inter-frame complementary information to improve the clarity, structural integrity, and visual quality of the super-resolution results.

[0137] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps described in the above methods. The storage medium may be, for example, ROM / RAM, magnetic disk, optical disk, etc.

[0138] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A continuous shooting image super-resolution method based on discrete wavelet transform fusion, characterized in that, The method comprises the following steps: S 1, extracting the features of multiple low-resolution pictures based on an encoder, calculating the offset between the multiple low-resolution pictures and a base frame by using deformable convolution, aligning the picture features according to the offset, and obtaining the feature map of the low-resolution picture; S2, decomposing the feature map of the low-resolution picture into low-frequency components and high-frequency components with different directions by using a discrete wavelet transform technology, and obtaining the preliminary fusion features by fusing the wavelet features and the residual features; S3, separating the preliminary fusion feature channels into semantic branches and spatial branches based on a bidirectional guiding mechanism, and guiding the semantic features and the spatial features to complementarily enhance each other by using spatial attention and second-order channel attention, to obtain the fused feature map; S4, expanding the channels of the fused feature map by using a pixel shuffling algorithm, rearranging the space of the feature after the channel expansion, to convert the channel information into spatial information, and realizing efficient and fast up-sampling.

2. The method of claim 1, wherein, The step of extracting the features of multiple low-resolution pictures based on an encoder, calculating the offset between the multiple low-resolution pictures and a base frame by using deformable convolution, and aligning the picture features according to the offset to obtain the feature map of the low-resolution picture comprises the following steps: S 11, extracting the features of multiple low-resolution pictures by using an encoder to convert the pixel-level information into feature-level information; S 12, constructing a cascaded multi-scale deformable convolution alignment module according to the feature maps of the base frame and the current frame, and calculating the offset required by the deformable convolution of each current frame and base frame at different scale levels; S 13, sampling the current level feature map according to the offset, and calculating the offset estimation and feature sampling on each scale feature map by using the deformable convolution alignment module in combination with a preset cascade order; S 14, obtaining local features by using the receptive field of the low-level feature map, and capturing global features by using the receptive field of the high-level feature map, to realize rough-to-fine motion compensation, and obtain the feature map of the low-resolution picture after feature alignment.

3. The method of claim 2, wherein, The formula for calculating the offset is: ΔP k (p0) = Conv([f b , f c ]); The expression of the feature map of the low-resolution picture after feature alignment is: where ΔP k (p0) denotes the offset between the predicted current frame and the base frame, k denotes the sampling point, p0 denotes the current calculation position of the feature map, Conv denotes a convolution layer, f b denotes the base frame feature, f c denotes the current frame feature, Y o (p0) denotes the feature map of the low-resolution picture after feature alignment, K denotes that the low-resolution picture is at each position of the convolution kernel, w k denotes the convolution kernel weight at this position, p k denotes the relative coordinates within the convolution kernel, and f denotes the current convolution layer.

4. The method of claim 1, wherein, The step of decomposing the feature map of the low-resolution picture into low-frequency components and high-frequency components with different directions by using a discrete wavelet transform technology, and obtaining the preliminary fusion features by fusing the wavelet features and the residual features comprises the following steps: S21, decomposing the feature map of the low-resolution picture into a low-frequency component containing the low-frequency components of the image and four sub-bands containing the high-frequency components of the image edges and textures by using a discrete wavelet, to obtain the decomposed multi-component features; S22, splicing the decomposed multi-component features into a feature tensor by using a wavelet feature splicing mechanism, and performing convolution and activation function activation, to obtain high-dimensional features; S23, constructing a residual path, and reserving the original feature information by using a parallel convolution path, to alleviate the feature loss caused by wavelet transform; S24, performing residual addition on the wavelet features and the residual features as the output after the discrete wavelet transform fusion, to obtain the preliminary fusion features.

5. The method of claim 4, wherein, The four sub-bands include a low-frequency sub-band, a horizontal high-frequency sub-band, a vertical high-frequency sub-band and a diagonal high-frequency sub-band. Among them, the low-frequency sub-band is used to reflect the low-frequency component of the image, describe the overall contour and smooth area; the horizontal high-frequency sub-band is used to capture the horizontal edge and detail; the vertical high-frequency sub-band is used to extract the vertical edge information; and the diagonal high-frequency sub-band is used to reflect the edge characteristics and texture details in the diagonal line direction.

6. The method of claim 5, wherein, The expression of the low-frequency sub-band is: The expression of the horizontal high-frequency sub-band is: The expression of the vertical high-frequency sub-band is: The expression of the diagonal high-frequency sub-band is: where Y LL (i, j), Y LH (i, j), Y HL (i, j), Y HH (i, j) represent low frequency sub-band, horizontal high frequency sub-band, vertical high frequency sub-band and diagonal high frequency sub-band, respectively, and Y(i, j) represents the feature value of the aligned input image feature at index (i, j).

7. The method of claim 1, wherein, The step of separating the preliminary fusion feature channel into a semantic branch and a spatial branch to process high-level semantic information and low-level spatial details, respectively, based on the bidirectional guidance mechanism, and respectively using spatial attention and second-order channel attention to guide the complementary enhancement of semantic features and spatial features to obtain a fused feature map includes the following steps: S31, separating the preliminary fusion feature channel into a semantic branch and a spatial branch to process high-level semantic information and low-level spatial details, respectively; S32, using spatial attention to guide semantic features to focus on key spatial regions, and using second-order channel attention to guide spatial features to focus on key semantic channels to achieve complementary enhancement of the two kinds of features; S33, based on an adaptive feature allocation method, dynamically allocating the channel proportion of the main branch and the sub-branch to ensure that spatial details are not compressed while retaining sufficient semantic information, to obtain a fused feature map.

8. The method of claim 7, wherein, The step of using spatial attention to guide semantic features to focus on key spatial regions, and using second-order channel attention to guide spatial features to focus on key semantic channels includes: Using spatial attention to enhance the complement of spatial features to semantics to guide semantic features to focus on key spatial regions; Using second-order channel attention to enhance the guidance of semantics to space of the feature map to guide spatial features to focus on key semantic channels.

9. The method of claim 7, wherein, The second-order channel attention captures the high-order correlation between channels through second-order statistics to improve the feature expression capability, and realizes the global interaction between channels through eigenvalue decomposition of the covariance matrix to overcome the limitations of traditional first-order statistics.

10. The method of claim 1, wherein, The step of using a pixel shuffling algorithm to expand the channels of the fused feature map, and rearranging the space of the channel-expanded feature to convert channel information into spatial information to realize efficient and fast up-sampling includes the following steps: S41, using a convolution layer to expand the number of channels of the fused feature map to a preset multiple of the super-resolution multiple, and encoding the spatial information of the low-resolution picture into more channels to reserve information for subsequent spatial dimension expansion; S42, using a shuffling operation to decompose the channel index in the expanded high-channel feature map into the channel of the output image and the index for spatial rearrangement, and combining spatial mapping to map the corresponding positions of the input feature map to the spatial positions of the high-resolution image.

Images