Double-shot image super-resolution method based on feature matching and confidence guidance attention fusion
By employing a multi-candidate feature matching and confidence-guided attention fusion method, the problems of inaccurate feature matching and high computational complexity in dual-camera super-resolution are solved, achieving more efficient feature fusion and detail reconstruction results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-17
- Publication Date
- 2026-04-03
AI Technical Summary
Existing dual-camera super-resolution methods lack effective confidence assessment during feature matching and feature fusion, leading to incorrect matching and high computational complexity, which affects the quality and stability of super-resolution results, especially in areas with complex parallax, occlusion, and weak texture.
By employing a multi-candidate feature matching module and a confidence-guided attention fusion method, an index map and a weight map are generated by calculating cosine similarity. Combined with optical flow-guided deformable convolution alignment and confidence-adaptive modulation, accurate feature alignment and efficient fusion are achieved.
It improves the robustness and adaptability of feature matching, effectively suppresses false matching, enhances the accuracy and visual consistency of texture detail reconstruction, and reduces computational resource consumption.
Smart Images
Figure CN121788349A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of dual-camera image processing technology, specifically relating to a dual-camera image super-resolution method based on feature matching and confidence-guided attention fusion. Background Technology
[0002] Existing dual-camera super-resolution methods suffer from significant shortcomings in both feature matching and feature fusion processes. First, feature matching-based methods generally lack an effective mechanism for evaluating the confidence level of feature matching, failing to distinguish between reliable and unreliable matching results. When erroneous matches occur, these incorrect features are indiscriminately propagated to the final reconstructed image, leading to texture misalignment, structural distortion, and noticeable artifacts, severely impacting the quality and stability of the super-resolution results. Second, while attention-based methods can establish a global correlation between wide-angle and telephoto images at the feature level, their global attention computation complexity is extremely high, consuming enormous computational resources. Furthermore, the lack of spatial guidance in attention weight allocation prevents adaptive adjustment of attention intensity based on the matching quality of different regions, resulting in erroneous feature aggregation in occluded and weakly textured areas, thereby reducing the accuracy and visual consistency of detail restoration. Summary of the Invention
[0003] To achieve the above objectives, the technical solution of this invention is as follows: A dual-camera image super-resolution method based on feature matching and confidence-guided attention fusion. The main technical problem of this invention is: how to simultaneously improve the reliability of feature matching and the adaptability of feature fusion in dual-camera super-resolution tasks, and achieve robust processing of complex parallax, occlusion, and weak texture regions, thereby effectively improving the quality of texture detail reconstruction while ensuring structural preservation. The method includes the following steps:
[0004] Step 1: Wide-angle image feature extraction and upsampling: Extract wide-angle image features... Input wide-angle image encoder Extracting wide-angle image features The wide-angle image encoder It consists of three cascaded residual channel attention modules, each containing 8 residual units; the encoded wide-angle image features are upsampled. To obtain upsampled wide-angle features And use it as one of the inputs to the confidence-guided cross-attention module;
[0005] Step 2, Feature Encoding and Optical Flow Estimation: This involves encoding the telephoto image... Overlapping area image with wide-angle Input to lightweight residual encoder In the process, the corresponding telephoto image features are generated. Features of wide-angle overlapping regions Among them, lightweight residual encoders It includes four residual blocks; an optical flow estimation network is used. Estimating wide-angle overlapping region image and a telephoto image downsampled to the same scale by twice Optical flow between ;
[0006] Step 3, Optical Flow Guided Feature Alignment: Align the estimated optical flow obtained in Step 2 with the optical flow... and wide-angle overlapping region image features Upsampled to twice the scale of a telephoto image, the resulting upsampled features and optical flow and the telephoto image features obtained in step 2 Input to optically flow-guided deformable convolution module Output with wide-angle image Finely aligned telephoto image features ;
[0007] Step 4, Multi-candidate Feature Matching Module: Matching wide-angle images Overlapping area image with wide-angle Input VGG feature encoder Then, using a 3x3 window with a step size of 1, the obtained features are expanded into feature blocks, and the resulting wide-angle image feature blocks are used as queries. Wide-angle overlapping region feature blocks as keys To make a difference between the two in the channel dimension After normalization, the cosine similarity is calculated to obtain the cosine similarity matrix. ;
[0008] Based on the similarity calculation results, for each query location Select the three most similar key feature blocks and save them as an index map. The corresponding similarities are then weighted using a Softmax function with a temperature scaling of 0.07, and the weighted coefficients are saved as a weighted graph. The maximum similarity score corresponding to each query location is saved as a confidence graph. ;
[0009] Step 5, Confidence-guided deformation module: This module aligns the telephoto image features obtained in Step 3. Using a 9x9 window with a step size of 4 to expand into feature blocks as value features, for each query position... The index map is obtained from step 5. and the corresponding weighted graph The corresponding feature blocks are extracted from the value features and weighted summed to obtain aggregated value feature blocks. These aggregated value feature blocks are then folded and reconstructed according to the grid order to obtain the deformed telephoto image features. The output size is consistent with the target super-resolution image;
[0010] Step 6, High and Low Frequency Decomposition and Feature Modulation: The deformed telephoto image features obtained in Step 6 are... Decomposed into low-frequency components and high frequency part Low frequency part The high-frequency component was obtained by upsampling after 4×4 average pooling with a stride of 4. Original features Low-frequency characteristics The residuals will be used to construct the confidence plot. Upsampled to the telephoto feature scale and projected onto the channel-dimensional feature space, then compared metrically with the high-frequency features. Multiplying them yields the telephoto features after confidence modulation. ;
[0011] Step 7, Confidence-guided cross-attention module: The upsampled wide-angle image features obtained in Step 1... The modulated telephoto image features obtained in step 6 After dividing the image into 8x8 non-overlapping window sizes, the wide-angle image features are used. As a query, low-frequency telephoto image features As a key, to modulate telephoto image features As a value;
[0012] Next, the confidence graph obtained in step 4 will be used. Spatial position offset is obtained by flattening the window after learning linear projection. The obtained spatial position is offset The attention score is calculated using the query, key, and value as input. According to attention scores Weighted aggregation of value features;
[0013] Step 8, Feature Fusion and Image Reconstruction: Combine the aggregated feature values obtained in Step 7. Reconstruct the spatial feature map and generate the projected features. With learnable gating The gated attention output is combined with the features of the original wide-angle image. The images are stitched together and merged, then processed by a residual decoder to generate the final super-resolution image. ;
[0014] As an improvement to this invention, steps 2-3 use a deformable convolution alignment scheme based on optical flow guidance. First, the downsampled telephoto image... Overlapping area image with wide-angle Input to optical flow network Estimate the optical flow between them The expression is as follows:
[0015] (1),
[0016] The estimated optical flow is obtained Then, precise and detailed alignment is achieved through optical flow-guided deformable convolution, as shown in the following expression:
[0017] (2),
[0018] in, It is a feature of telephoto images. It is the optical flow after upsampling. These are the features of the wide-angle overlapping region image after upsampling. It is a deformable convolution, with each feature having 64 channels. Through optical flow-guided deformable convolution, telephoto image features are obtained that are precisely aligned to the wide-angle image region. .
[0019] As an improvement to this invention, the multi-candidate feature matching module and confidence-guided deformation module proposed in steps 4-5 are used. Input wide-angle image. Overlapping area image with wide-angle After VGG feature encoder To extract features, use a 3x3 window with a stride of 1 to expand the extracted features into feature blocks, as shown in the following expression:
[0020] (3),
[0021] (4),
[0022] in The operation involves cutting a whole feature map into many small blocks using a sliding window, and then flattening each block into a column. and These are wide-angle images Overlapping images with wide-angle lenses The feature blocks are expanded, and then the column vectors are normalized to calculate the cosine similarity, as shown in the following expression:
[0023] (5),
[0024] in For each query position, the cosine similarity matrix is... Choosing the three most similar key positions can be represented as:
[0025] (6),
[0026] in Indicates the first The first wide-angle image feature block and the second The cosine similarity of the feature blocks in the overlapping regions, with a similarity range of [value missing]. , Representation and query block The set of indices of the three most similar key feature blocks is output as an index map, which is then scaled by temperature. The softmax algorithm calculates the weighting coefficients using the following expression:
[0027] (7),
[0028] and confidence level Calculated This represents the Top 3 weight stack, recording the weights of the top three similar feature blocks obtained after weighting. Confidence level. Represents the first The matching confidence score for each query block location is used, and the highest similarity score is taken; the closer the value is to 1, the more reliable it is. Then, in the aligned telephoto image features... Perform index-based weighted patch extraction and reconstruction to extract and reconstruct telephoto features. Expanded into a block sequence, used as a value feature, the expression is as follows:
[0029] (8),
[0030] in This indicates a 9x9 window with a step size of 4. Operation, query the location for each wide-angle image. ,according to Corresponding index, from value feature Extract the corresponding feature blocks from the data and sort them by weight. To perform aggregation, the expression is as follows:
[0031] (9),
[0032] in The feature of a telephoto image is represented by the first... The feature vectors at each position are weighted and aggregated using the three most similar feature blocks and their corresponding weights, smoothly fusing information from multiple perspectives. The aggregated column vectors at all positions... The concatenation expression is as follows:
[0033] (10)
[0034] Finally, folding and reconstruction are performed to obtain the deformed telephoto image features. , can be represented as:
[0035] (11),
[0036] in express Operation, and Maintaining the same window size and stride, the features of the deformed telephoto image... It is already the target super-resolution image resolution.
[0037] As an improvement to this invention, a confidence-guided cross-attention module is proposed in steps 6-8. First, the features of the deformed telephoto image are... It is decomposed into low-frequency and high-frequency components. The low-frequency components can be obtained by upsampling after average pooling, as shown in the following expression:
[0038] (12)
[0039] in That is, the low-frequency components of the features of telephoto images. This represents average pooling with a size of 4×4 and a stride of 4. This represents 4x double upsampling. Then, the high-frequency residual is calculated and expressed as:
[0040] (13)
[0041] in These are the high-frequency components of telephoto image features. This is a learnable scaling parameter initialized to 1.0, allowing the network to adaptively control the intensity of high-frequency injections during training. A factor of 2 is set to compensate for energy loss during frequency separation. The confidence plot of the feature matching module obtained in the previous step... This can represent the reliability of the correspondence at each spatial location. The confidence information is upsampled to the telephoto feature scale and projected onto the channel-dimensional feature space, as shown in the following expression:
[0042] (14)
[0043] in It consists of two convolutional layers with LeakyReLU activation. The first layer projects from 1 channel to 16 channels using a 7×7 kernel to obtain the spatial context, while the second layer projects to 64 channels using a 3×3 kernel. Modulated injected features It can be represented as:
[0044] (15)
[0045] in This indicates element-wise multiplication; modulation suppresses unreliable details, ensuring that high-frequency details can be transferred to regions with high confidence levels. The input is modulated telephoto image features. and upsampled wide-angle image features Divided into sizes of 8 There are a total of non-overlapping local windows. One window, by head Grouping, for each window The query is calculated using wide-angle image features, the key is calculated using low-frequency telephoto distortion features, and the value is calculated using modulated injection features. The expression is as follows:
[0046] (16)
[0047] in, It is a linear projection. This is the m-th window containing the calculated query, key, and value features. For each header and window, an additive bias is calculated using the confidence plot, expressed as follows:
[0048] (17)
[0049] in It is a learnable 1×1 convolution (linear projection) that performs head-by-head 1×1 convolution projection on the confidence map, and then obtains the spatial offset by dividing and flattening it according to the window. Next, the attention score is calculated, as shown in the following expression:
[0050] (18)
[0051] in It is a hyperparameter used to control the confidence bias strength. Ensure bias is limited To prevent excessive influence caused by excessive bias within the range, The attention score is the value of the m-th window. The calculated attention score is used to perform weighted aggregation on the value features, as shown in the following expression:
[0052] (19)
[0053] Received This refers to the attention output (weighted aggregation of value features) of the m-th window. After calculating the attention outputs of all windows and the head, the spatial feature map is reconstructed, the outputs within the window are inversely segmented and pieced back together, and then the projected features and learnable gating are output. This process can be represented as:
[0054] (20)
[0055] in To output the projection matrix, It is the concatenated global attention output feature map. It outputs the projected global attention feature map. For learnable gated projection matrices, It is the Sigmoid activation function. It is a learnable gated graph. The gating mechanism generates weights between 0 and 1 for each spatial location and channel. The closer the weight is to 1, the more reliable the feature. Finally, the attention output after gating is used. Features of the original wide-angle image The expression for concatenating and merging is as follows:
[0056] (twenty one),
[0057] in It is a 3×3 convolutional layer with LeakyReLU activation, which reduces the channel dimension from 128 to 64. Indicates splicing along the channel. This represents element-wise multiplication. The fused features are processed by the residual decoder. The process generates the final super-resolution image, represented as:
[0058] (twenty two),
[0059] in It consists of 4 residual blocks, and the final convolution projects the features back to the RGB space.
[0060] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0061] 1) This invention designs a multi-candidate feature matching module and proposes a confidence evaluation scheme. By calculating cosine similarity, an index map, a weight map, and a confidence map are generated. The top three most similar feature blocks are selected and weighted and aggregated based on the index map and the weight map, which effectively improves the robustness and accuracy of feature matching and effectively suppresses false matching under the condition of weak similarity and resolution differences.
[0062] 2) This invention designs a confidence-guided cross-attention module, which not only utilizes the computational efficiency advantage of local window attention, but also innovatively proposes a fusion scheme that combines confidence bias guidance and frequency decomposition. By projecting the confidence map into a learnable spatial bias and embedding it into the attention calculation, and using a frequency decomposition strategy to separate telephoto features into high and low frequency components, and using confidence-adaptive modulation of high frequency transmission intensity, a balance is effectively achieved between structure preservation and detail enhancement. Differentiated adaptive processing is achieved in different confidence regions, which ensures accurate transmission of texture details in high confidence regions and suppresses artifacts in low confidence regions. Attached Figure Description
[0063] Figure 1 This is a diagram of the algorithm framework of the present invention;
[0064] Figure 2 This is a schematic diagram of the multi-candidate feature matching module structure of the present invention;
[0065] Figure 3 This is a schematic diagram of the confidence-guided deformation module structure of the present invention;
[0066] Figure 4 This is a schematic diagram of the confidence-guided cross-attention module structure of the present invention. Detailed Implementation
[0067] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.
[0068] Example: Figure 1 As shown, this embodiment provides a dual-camera image super-resolution method based on feature matching and confidence-guided attention fusion, specifically including the following steps:
[0069] Step 1: Wide-angle image feature extraction and upsampling: Extract wide-angle image features... Input wide-angle image encoder Extracting wide-angle image features The wide-angle image encoder It consists of three cascaded residual channel attention modules, each containing 8 residual units; the encoded wide-angle image features are upsampled. To obtain upsampled wide-angle features And use it as one of the inputs to the confidence-guided cross-attention module;
[0070] Step 2, Feature Encoding and Optical Flow Estimation: This involves encoding the telephoto image... Overlapping area image with wide-angle Input to lightweight residual encoder In the process, the corresponding telephoto image features are generated. Features of wide-angle overlapping regions Among them, lightweight residual encoders It includes four residual blocks; an optical flow estimation network is used. Estimating wide-angle overlapping region image and a telephoto image downsampled to the same scale by twice Optical flow between ;
[0071] Step 3, Optical Flow Guided Feature Alignment: Align the estimated optical flow obtained in Step 2 with the optical flow... and wide-angle overlapping region image features Upsampled to twice the scale of a telephoto image, the resulting upsampled features and optical flow and the telephoto image features obtained in step 2 Input to optically flow-guided deformable convolution module Output with wide-angle image Finely aligned telephoto image features ;
[0072] Step 4, Multi-candidate Feature Matching Module: Matching wide-angle images Overlapping area image with wide-angle Input VGG feature encoder Then, using a 3x3 window with a step size of 1, the obtained features are expanded into feature blocks, and the resulting wide-angle image feature blocks are used as queries. Wide-angle overlapping region feature blocks as keys To make a difference between the two in the channel dimension After normalization, the cosine similarity is calculated to obtain the cosine similarity matrix. ;
[0073] Based on the similarity calculation results, for each query location Select the three most similar key feature blocks and save them as an index map. The corresponding similarities are then weighted using a Softmax function with a temperature scaling of 0.07, and the weighted coefficients are saved as a weighted graph. The maximum similarity score corresponding to each query location is saved as a confidence graph. ;
[0074] Step 5, Confidence-guided deformation module: This module aligns the telephoto image features obtained in Step 3. Using a 9x9 window with a step size of 4 to expand into feature blocks as value features, for each query position... The index map is obtained from step 5. and the corresponding weighted graph The corresponding feature blocks are extracted from the value features and weighted summed to obtain aggregated value feature blocks. These aggregated value feature blocks are then folded and reconstructed according to the grid order to obtain the deformed telephoto image features. The output size is consistent with the target super-resolution image;
[0075] Step 6, High and Low Frequency Decomposition and Feature Modulation: The deformed telephoto image features obtained in Step 6 are... Decomposed into low-frequency components and high frequency part Low frequency part The high-frequency component was obtained by upsampling after 4×4 average pooling with a stride of 4. Original features Low-frequency characteristics The residuals will be used to construct the confidence plot. Upsampled to the telephoto feature scale and projected onto the channel-dimensional feature space, then compared metrically with the high-frequency features. Multiplying them yields the telephoto features after confidence modulation. ;
[0076] Step 7, Confidence-guided cross-attention module: The upsampled wide-angle image features obtained in Step 1... The modulated telephoto image features obtained in step 6 After dividing the image into 8x8 non-overlapping window sizes, the wide-angle image features are used. As a query, low-frequency telephoto image features As a key, to modulate telephoto image features As a value;
[0077] Next, the confidence graph obtained in step 4 will be used. Spatial position offset is obtained by flattening the window after learning linear projection. The obtained spatial position is offset The attention score is calculated using the query, key, and value as input. According to attention scores Weighted aggregation of value features;
[0078] Step 8, Feature Fusion and Image Reconstruction: Combine the aggregated feature values obtained in Step 7. Reconstruct the spatial feature map and generate the projected features. With learnable gating The gated attention output is combined with the features of the original wide-angle image. The images are stitched together and merged, then processed by a residual decoder to generate the final super-resolution image. ;
[0079] As an improvement to this invention, steps 2-3 use a deformable convolution alignment scheme based on optical flow guidance. First, the downsampled telephoto image... Overlapping area image with wide-angle Input to optical flow network Estimate the optical flow between them The expression is as follows:
[0080] (1),
[0081] The estimated optical flow is obtained Then, precise and detailed alignment is achieved through optical flow-guided deformable convolution, as shown in the following expression:
[0082] (2),
[0083] in, It is a feature of telephoto images. It is the optical flow after upsampling. These are the features of the wide-angle overlapping region image after upsampling. It is a deformable convolution, with each feature having 64 channels. Through optical flow-guided deformable convolution, telephoto image features are obtained that are precisely aligned to the wide-angle image region. .
[0084] As an improvement to this invention, the multi-candidate feature matching module and confidence-guided deformation module proposed in steps 4-5 are used. Input wide-angle image. Overlapping area image with wide-angle After VGG feature encoder To extract features, use a 3x3 window with a stride of 1 to expand the extracted features into feature blocks, as shown in the following expression:
[0085] (3),
[0086] (4),
[0087] in The operation involves cutting a whole feature map into many small blocks using a sliding window, and then flattening each block into a column. and These are wide-angle images Overlapping images with wide-angle lenses The feature blocks are expanded, and then the column vectors are normalized to calculate the cosine similarity, as shown in the following expression:
[0088] (5),
[0089] in For each query position, the cosine similarity matrix is... Choosing the three most similar key positions can be represented as:
[0090] (6),
[0091] in Indicates the first The first wide-angle image feature block and the second The cosine similarity of the feature blocks in the overlapping regions, with a similarity range of [value missing]. , Representation and query block The set of indices of the three most similar key feature blocks is output as an index map, which is then scaled by temperature. The softmax algorithm calculates the weighting coefficients using the following expression:
[0092] (7),
[0093] and confidence level Calculated This represents the Top 3 weight stack, recording the weights of the top three similar feature blocks obtained after weighting. Confidence level. Represents the first The matching confidence score for each query block position is used, and the highest similarity score is taken; the closer the value is to 1, the more reliable it is. The specific architecture of the multi-candidate feature matching module is as follows: Figure 2 As shown. Next, in the aligned telephoto image features... Perform index-based weighted patch extraction and reconstruction to extract and reconstruct telephoto features. Expanded into a block sequence, used as a value feature, the expression is as follows:
[0094] (8),
[0095] in This indicates a 9x9 window with a step size of 4. Operation, query the location for each wide-angle image. ,according to Corresponding index, from value feature Extract the corresponding feature blocks from the data and sort them by weight. To perform aggregation, the expression is as follows:
[0096] (9),
[0097] in The feature of a telephoto image is represented by the first... The feature vectors at each position are weighted and aggregated using the three most similar feature blocks and their corresponding weights, smoothly fusing information from multiple perspectives. The aggregated column vectors at all positions... The concatenation expression is as follows:
[0098] (10)
[0099] Finally, folding and reconstruction are performed to obtain the deformed telephoto image features. , can be represented as:
[0100] (11),
[0101] in express Operation, and Maintaining the same window size and stride, the features of the deformed telephoto image... The target super-resolution image resolution has been reached. The specific architecture of the confidence-guided deformation module is as follows: Figure 3 As shown.
[0102] As an improvement to this invention, a confidence-guided cross-attention module is proposed in steps 6-8. First, the features of the deformed telephoto image are... It is decomposed into low-frequency and high-frequency components. The low-frequency components can be obtained by upsampling after average pooling, as shown in the following expression:
[0103] (12)
[0104] in That is, the low-frequency components of the features of telephoto images. This represents average pooling with a size of 4×4 and a stride of 4. This represents a quadruple upsampling (4 times). Then, the high-frequency residual is calculated and expressed as:
[0105] (13)
[0106] in These are the high-frequency components of telephoto image features. This is a learnable scaling parameter initialized to 1.0, allowing the network to adaptively control the intensity of high-frequency injections during training. A factor of 2 is set to compensate for energy loss during frequency separation. The confidence plot of the feature matching module obtained in the previous step... This can represent the reliability of the correspondence at each spatial location. The confidence information is upsampled to the telephoto feature scale and projected onto the channel-dimensional feature space, as shown in the following expression:
[0107] (14)
[0108] in It consists of two convolutional layers with LeakyReLU activation. The first layer projects from 1 channel to 16 channels using a 7×7 kernel to obtain the spatial context, while the second layer projects to 64 channels using a 3×3 kernel. Modulated injected features It can be represented as:
[0109] (15)
[0110] in This indicates element-wise multiplication; modulation suppresses unreliable details, ensuring that high-frequency details can be transferred to regions with high confidence levels. The input is modulated telephoto image features. and upsampled wide-angle image features Divided into sizes of 8 There are a total of non-overlapping local windows. One window, by head Grouping, for each window The query is calculated using wide-angle image features, the key is calculated using low-frequency telephoto distortion features, and the value is calculated using modulated injection features. The expression is as follows:
[0111] (16)
[0112] in, It is a linear projection. This is the m-th window containing the calculated query, key, and value features. For each header and window, an additive bias is calculated using the confidence plot, expressed as follows:
[0113] (17)
[0114] in It is a learnable 1×1 convolution (linear projection) that performs head-by-head 1×1 convolution projection on the confidence map, and then obtains the spatial offset by dividing and flattening it according to the window. Next, the attention score is calculated, as shown in the following expression:
[0115] (18)
[0116] in It is a hyperparameter used to control the confidence bias strength. Ensure bias is limited To prevent excessive influence caused by excessive bias within the range, The attention score is the value of the m-th window. The calculated attention score is used to perform weighted aggregation on the value features, as shown in the following expression:
[0117] (19)
[0118] Received This refers to the attention output (weighted aggregation of value features) of the m-th window. After calculating the attention outputs of all windows and the head, the spatial feature map is reconstructed, the outputs within the window are inversely segmented and pieced back together, and then the projected features and learnable gating are output. This process can be represented as:
[0119] (20)
[0120] in To output the projection matrix, It is the concatenated global attention output feature map. It outputs the projected global attention feature map. For learnable gated projection matrices, It is the Sigmoid activation function. It is a learnable gated graph. The gating mechanism generates weights between 0 and 1 for each spatial location and channel. The closer the weight is to 1, the more reliable the feature. Finally, the attention output after gating is used. Features of the original wide-angle image The expression for concatenating and merging is as follows:
[0121] (twenty one),
[0122] in It is a 3×3 convolutional layer with LeakyReLU activation, which reduces the channel dimension from 128 to 64. Indicates splicing along the channel. This represents element-wise multiplication. The fused features are processed by the residual decoder. The process generates the final super-resolution image, represented as:
[0123] (twenty two),
[0124] in Composed of four residual blocks, the final convolution projects the features back into the RGB space. The specific architecture of the confidence-guided cross-attention module is as follows: Figure 4 As shown.
[0125] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.
Claims
1. A dual-camera image super-resolution method based on feature matching and confidence-guided attention fusion, characterized in that, Step 1: Capture the wide-angle image Input wide-angle image encoder Extracting wide-angle image features Upsampling of the encoded wide-angle image features To obtain upsampled wide-angle features ; Step 2: Capture the telephoto image Overlapping area image with wide-angle Input to lightweight residual encoder In the process, the corresponding telephoto image features are generated. Features of wide-angle overlapping regions An optical flow estimation network is used. Estimating wide-angle overlapping region image and a telephoto image downsampled to the same scale by twice Optical flow between ; Step 3: The optical flow obtained in Step 2 and wide-angle overlapping region image features Upsampled to twice the scale of a telephoto image, the resulting upsampled features and upsampling optical flow and the telephoto image features obtained in step 2 Input to optically flow-guided deformable convolution module Output with wide-angle image Finely aligned telephoto image features ; Step 4: Capture the wide-angle image Overlapping area image with wide-angle Input VGG feature encoder The obtained wide-angle image feature blocks and wide-angle overlapping region feature blocks are then processed in the channel dimension. After normalization, the cosine similarity matrix is obtained. For each query position The index map is generated by selecting the three most similar key feature blocks. The corresponding similarities are converted into weighted coefficients using the Softmax function to obtain the weighted graph. The maximum similarity score corresponding to each query location is saved as a confidence graph. ; Step 5: Align the telephoto image features As a value feature, for each query position The corresponding feature blocks are extracted from the value features and weighted summed to obtain aggregated value feature blocks. These aggregated value feature blocks are then folded and reconstructed according to the grid order to obtain the deformed telephoto image features. ; Step 6: Extract features from the distorted telephoto image. Decomposed into low-frequency components and high frequency part Low frequency part The confidence map was obtained by upsampling after average pooling. Upsampled to the telephoto feature scale and projected onto the channel-dimensional feature space, then compared metrically with the high-frequency features. Multiplying them yields the telephoto features after confidence modulation. ; Step 7: Upsample wide-angle image features and the features of the telephoto image after modulation After non-overlapping partitioning, the confidence graph is... Spatial position offset is obtained by flattening the window after learning linear projection. The obtained spatial position is offset Attention score is calculated as input. According to attention scores Weighted aggregation of value features; Step 8: Aggregate value features Reconstruct the spatial feature map and generate the projected features. With learnable gating The gated attention output is combined with the features of the original wide-angle image. The images are stitched together and merged, then processed by a residual decoder to generate the final super-resolution image. .
2. The method according to claim 1, characterized in that, In step 4, based on the similarity calculation results, for each query location... Select the three most similar key feature blocks and save them as an index map. The corresponding similarities are then weighted using a Softmax function with a temperature scaling of 0.07, and the weighted coefficients are saved as a weighted graph. The maximum similarity score corresponding to each query location is saved as a confidence graph. .
3. The method according to claim 1, characterized in that: Steps 2-3 use a deformable convolution alignment scheme guided by optical flow. First, the downsampled telephoto image is... Overlapping area image with wide-angle Input to optical flow network Estimate the optical flow between them The expression is as follows: （1）， The estimated optical flow is obtained Then, precise and detailed alignment is achieved through optical flow-guided deformable convolution, as shown in the following expression: （2）， in, It is a feature of telephoto images. It is the optical flow after upsampling. These are the features of the wide-angle overlapping region image after upsampling. It is a deformable convolution, with each feature having 64 channels. Through optical flow-guided deformable convolution, telephoto image features are obtained that are precisely aligned to the wide-angle image region. .
4. The method according to claim 1, characterized in that: The multi-candidate feature matching module and confidence-guided deformation module proposed in steps 4-5 are used as input wide-angle images. Overlapping area image with wide-angle After VGG feature encoder Extract features, and expand the extracted features into feature blocks, as shown in the following expression: （3）， （4）， in The operation involves cutting a whole feature map into many small blocks using a sliding window, and then flattening each block into a column. and These are wide-angle images Overlapping images with wide-angle lenses The feature blocks are expanded, and then the column vectors are normalized to calculate the cosine similarity, as shown in the following expression: （5）， in For each query position, the cosine similarity matrix is... Select the three most similar key positions, represented as: （6）， in Indicates the first The first wide-angle image feature block and the second The cosine similarity of the feature blocks in the overlapping regions, with a similarity range of [value missing]. , Representation and query block The set of indices of the three most similar key feature blocks is output as an index map, which is then scaled by temperature. The softmax algorithm calculates the weighting coefficients using the following expression: （7）， and confidence level Calculated This represents the Top 3 weight stack, recording the weights and confidence scores of the top three similar feature blocks obtained after weighting. Represents the first The matching confidence of each query block location is used, and the highest similarity is taken. The closer the value is to 1, the more reliable it is. Then, the aligned telephoto image features are analyzed. Perform index-based weighted patch extraction and reconstruction to extract and reconstruct telephoto features. Expanded into a block sequence, used as a value feature, the expression is as follows: （8）， in This indicates a 9x9 window with a step size of 4. Operation, query the location for each wide-angle image. ,according to Corresponding index, from value feature Extract the corresponding feature blocks from the data and sort them by weight. To perform aggregation, the expression is as follows: （9）， in The feature of a telephoto image is represented by the first... The feature vectors at each position are weighted and aggregated using the three most similar feature blocks and their corresponding weights, smoothly fusing information from multiple perspectives. The resulting column vectors at all positions are then processed. The concatenation expression is as follows: （10）， Finally, folding and reconstruction are performed to obtain the deformed telephoto image features. , can be represented as: （11）， in express Operation, and Maintaining the same window size and stride, the features of the deformed telephoto image... It is already the target super-resolution image resolution.
5. The method according to claim 1, characterized in that: The confidence-guided cross-attention module proposed in steps 6-8 first integrates the features of the deformed telephoto image. It is decomposed into low-frequency and high-frequency components. The low-frequency components can be obtained by upsampling after average pooling, as shown in the following expression: （12）， in That is, the low-frequency components of the features of telephoto images. This represents average pooling with a size of 4×4 and a stride of 4. This indicates a 4x double cubic upsampling, followed by calculation of the high-frequency residual, expressed as: （13）， in These are the high-frequency components of telephoto image features. is a learnable scaling parameter initialized to 1.0, allowing the network to adaptively control the intensity of high-frequency injection during training. A factor of 2 is set to compensate for energy loss during frequency separation. The confidence plot of the feature matching module obtained in the previous step is also shown. To represent the reliability of the correspondence at each spatial location, the confidence information is upsampled to the telephoto feature scale and projected onto the channel-dimensional feature space, as shown in the following expression: （14）， in It consists of two convolutional layers with LeakyReLU activation. The first layer uses a 7×7 kernel to project from 1 channel to 16 channels to obtain the spatial context, while the second layer uses a 3×3 kernel to project to 64 channels to obtain the modulated injected features. It can be represented as: （15）， in This represents element-wise multiplication, which takes the input modulated telephoto image features as input. and upsampled wide-angle image features Divided into sizes of 8 There are a total of non-overlapping local windows. One window, by head Grouping, for each window The query is calculated using wide-angle image features, the key is calculated using low-frequency telephoto distortion features, and the value is calculated using modulated injection features. The expression is as follows: （16）， in, It is a linear projection. This is the m-th window containing the calculated query, key, and value features. For each header and window, an additive bias is calculated using the confidence plot, expressed as follows: （17）， in It is a learnable 1×1 convolution that performs a head-by-head 1×1 convolution projection on the confidence map, and then divides and flattens it according to the window to obtain the spatial offset. Next, the attention score is calculated, as shown in the following expression: （18）， in It is a hyperparameter used to control the confidence bias strength. Ensure bias is limited To prevent excessive influence caused by excessive bias within the range, The attention score is the value of the m-th window. The calculated attention score is used to perform weighted aggregation on the value features, as shown in the following expression: （19）， Received This refers to the attention output of the m-th window. After calculating the attention outputs of all windows and the head, the spatial feature map is reconstructed, the outputs within the window are inversely segmented and pieced back together, and then the projected features and learnable gating are output. This process can be represented as: （20）， in To output the projection matrix, It is the concatenated global attention output feature map. It outputs the projected global attention feature map. For learnable gated projection matrices, It is the Sigmoid activation function. It is a learnable gated graph. The gating mechanism generates weights between 0 and 1 for each spatial location and channel. The closer the weight is to 1, the more reliable the feature. Finally, the attention output after gating is used. Features of the original wide-angle image The expression for concatenating and merging is as follows: （21）， in It is a 3×3 convolutional layer with LeakyReLU activation, which reduces the channel dimension from 128 to 64. Indicates splicing along the channel. This represents element-wise multiplication, with fused features processed by a residual decoder. The process generates the final super-resolution image, represented as: （22）， in It consists of 4 residual blocks, and the final convolution projects the features back to the RGB space.