Dual self-synchronization based super-resolution zoom reconstruction method and system

Abstract

This invention relates to the field of image processing, specifically disclosing a super-resolution zoom reconstruction method based on dual self-coordination, comprising the following steps: S1 Image acquisition: capturing a low-resolution image to be reconstructed using a wide-angle lens, and capturing a high-resolution reference image of the same scene using a telephoto lens; S2 Feature extraction: inputting the image and reference image into a feature extraction module to extract a low-resolution feature map and a reference feature map containing low-level local features and high-level semantic features; S3 Feature matching and region discrimination; S4 Dual-branch feature reconstruction; S4-1 Overlapping region feature alignment; S4-2 Non-overlapping region feature enhancement; S5 Dual-stream feature fusion; S6 Super-resolution reconstruction. The technical solution of this invention can simultaneously achieve high-precision alignment reconstruction of overlapping regions and reasonable semantic reconstruction of non-overlapping regions, and can achieve a smooth transition between the reconstruction effects of the two regions, achieving high-quality, high-consistency full-image super-resolution reconstruction.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method and system for super-resolution zoom reconstruction based on dual self-coordination. Background Technology

[0002] With the rapid development of mobile terminal imaging technology, asymmetric multi-camera modules have become the mainstream hardware configuration for smartphones and other mobile terminals. These modules typically include a wide-angle main camera and a telephoto lens, providing a hardware foundation for the implementation of computational photography technology. Based on this hardware architecture, reference image super-resolution technology has emerged. This technology uses a high-resolution image captured by the telephoto lens as a reference to enhance and reconstruct a low-resolution image captured by the wide-angle lens. Compared to traditional single-image super-resolution technology, which relies solely on prior knowledge learned from a large amount of training data to construct high-frequency details, reference image super-resolution technology can utilize the real texture information existing in the reference image within the same scene, effectively improving the visual realism and detail richness of the reconstructed image, thus becoming one of the core technologies for dual-camera zoom imaging in mobile terminals.

[0003] Current mainstream reference image super-resolution methods focus on solving the spatial misalignment problem between input and reference images. Due to the parallax in the mounting positions of the two cameras in a dual-camera module and the significant difference in zoom magnification, complex geometric distortions often exist between wide-angle low-resolution images and telephoto high-resolution reference images. To address this problem, existing technologies mainly fall into two categories: one is explicit alignment methods based on optical flow or deformable convolution, which estimate pixel-level offset fields to align the features of the reference image to the feature space of the input image, achieving cross-image texture transfer; the other is implicit alignment methods based on block matching or attention mechanisms, which search for similar texture blocks between the input and reference images in the feature space and achieve image enhancement through texture block fusion. Currently, some advanced reference image super-resolution algorithms combine multi-scale feature extraction and deformable convolutional networks to achieve accurate cross-scale texture transfer, achieving good reconstruction results in the overlapping field of view regions of wide-angle and telephoto images.

[0004] However, existing reference image super-resolution technologies still suffer from insurmountable technical defects in practical dual-camera zoom applications, failing to meet the actual needs of high-quality full-field-of-view imaging for mobile terminals. The core problem lies in the inability of existing technologies to effectively handle the reconstruction of non-overlapping regions caused by the difference in field of view between wide-angle and telephoto lenses. Specifically, the field of view of a telephoto lens is much smaller than that of a wide-angle lens. The edge regions of the low-resolution image captured by the wide-angle lens have no corresponding texture information in the high-resolution reference image captured by the telephoto lens. Moreover, most existing reference image super-resolution algorithms use globally consistent feature processing and image reconstruction logic, failing to distinguish between overlapping and non-overlapping regions in the image. When aligning features in non-overlapping regions, the lack of corresponding texture information in the reference image leads to the forced alignment operation incorrectly matching irrelevant texture features in the reference image to the non-overlapping regions. This results in obvious ghosting, texture distortion, geometric distortion, and other artifacts in the edge regions of the reconstructed image, severely degrading the imaging quality of the edge regions.

[0005] To avoid the aforementioned mismatch problem, some simple improvement strategies have emerged in existing technologies, namely, directly degrading to single-image super-resolution algorithms or traditional interpolation algorithms for image reconstruction in non-overlapping areas. However, such strategies will cause new technical problems: there is a hard switch in the reconstruction logic between overlapping and non-overlapping areas, resulting in obvious visual boundaries at the junction of the two areas in the reconstructed image. Furthermore, there is a huge resolution gap between the image quality of the non-overlapping area and the central overlapping area. The signal-to-noise ratio and texture style of the two areas cannot be kept consistent, which seriously undermines the user's visual consistency experience across the entire field of view.

[0006] In summary, existing technologies lack a dual-self-cooperative super-resolution zoom reconstruction method and system that can simultaneously achieve high-precision alignment reconstruction of overlapping regions and semantically reasonable reconstruction of non-overlapping regions, and can smoothly transition the reconstruction effects of the two regions, thus achieving high-quality, highly consistent full-image super-resolution reconstruction. Summary of the Invention

[0007] This invention provides a super-resolution zoom reconstruction method based on dual self-cooperation, which can simultaneously take into account high-precision alignment reconstruction of overlapping areas and semantically reasonable reconstruction of non-overlapping areas, and can achieve a smooth transition of reconstruction effects between the two areas, thus achieving high-quality and highly consistent full-image super-resolution reconstruction.

[0008] To solve the above-mentioned technical problems, this application provides the following technical solution: The super-resolution zoom reconstruction method based on dual self-cooperation includes the following steps: S1 Image Acquisition: Captures low-resolution images to be reconstructed using a wide-angle lens. Capture high-resolution reference images of the same scene using a telephoto lens. ; S2 Feature Extraction: and The input feature extraction module extracts a low-resolution feature map containing both low-level local features and high-level semantic features. and reference feature map The feature extraction module is implemented using a residual convolutional network, a visual Transformer, or a lightweight network architecture. S3 Feature Matching and Region Determination: [The sentence is incomplete and likely refers to a specific feature matching and region discrimination method.] and The input feature matching module uses sliding window technology to expand the two into a sequence of local feature blocks and performs L2 normalization. It calculates the cosine similarity matrix through matrix multiplication, and then performs maximum search and parameter search in the reference feature dimension to generate a matching confidence map C and an index map P. The matching confidence map C accurately distinguishes the overlapping and non-overlapping regions of the image to be reconstructed, and the index map P records the spatial coordinates of the most similar texture in the reference feature map. S4 dual-branch feature reconstruction: S4-1 Overlapping Region Feature Alignment: Based on Index Graph P Pre-distorted feature maps are obtained by performing block-based hard distortion. Then, through a learnable and deformable alignment module... Fine alignment is performed to obtain the alignment feature map. The deformable alignment module is implemented using a global alignment strategy based on deformable convolution, optical flow estimation, or affine / homography transformation. S4-2 Non-overlapping region feature enhancement: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Perform multi-scale context aggregation to obtain a multi-scale reference feature library. ,Will Mapped to query vector Q, Mapped to key vector K and value vector V respectively, attention map A is generated through cross-attention mechanism, and non-overlapping region mask is constructed by combining it with matching confidence map C. After filtering the output of attention map A, it is compared with... Weighted recombination yields enhanced feature maps of non-overlapping regions. ; S5 Dual-Stream Feature Fusion: and The input is processed by an adaptive dual-stream fusion module, which employs a dynamic selection fusion strategy based on confidence-based linear weighting, channel convolutional fusion, or gating mechanisms to perform feature-level adaptive fusion, resulting in a full-field fused feature map. This enables a smooth transition between overlapping and non-overlapping regions. S6 Super-Resolution Reconstruction: The input reconstruction module employs upsampling strategies combining subpixel convolution, deconvolution, or interpolation with post-processing to map the low-resolution feature space back to the high-resolution image space, outputting a full-field high-resolution super-resolution reconstructed image.

[0009] The basic principles and beneficial effects of the scheme are as follows: This invention takes a confidence-guided dual-stream adaptive fusion architecture as its core and addresses the inherent problem of inconsistent field of view between wide-angle and telephoto lenses in asymmetric dual-camera zoom systems. It breaks down the image reconstruction process into three key steps: region discrimination, branch reconstruction, and adaptive fusion, thereby achieving the unity of precise alignment of overlapping areas, semantic illusion reconstruction of non-overlapping areas, and smooth transition of the entire image in principle.

[0010] The principle of precise region discrimination constructs a matching confidence map C in the feature space through sliding window segmentation, L2 normalization, and cosine similarity calculation. It objectively distinguishes overlapping regions with reference textures from non-overlapping regions without reference textures by numerical value, replacing the global unified processing logic of existing technologies and avoiding artifacts caused by forced alignment of non-overlapping regions from the source.

[0011] Overlapping areas are aligned in a coarse-to-fine hierarchical manner. First, the index map P is used to complete the block-level coarse distortion, and then the learnable alignment module is used to eliminate geometric misalignment and completely transfer the high-frequency texture of the reference map to ensure high-precision detail restoration.

[0012] For non-overlapping regions, multi-scale cross-attention enhancement is used. First, multi-scale context is aggregated to form a semantic feature library. Then, similar texture patterns are retrieved through cross-spatial attention to complete reasonable semantic illusion construction without pixel correspondence.

[0013] The seamless smooth fusion principle uses the confidence map as a dynamic weight to adaptively weight and fuse the two outputs in the feature space, rather than hard stitching the image layers. This allows the overlapping and non-overlapping areas to transition continuously in terms of texture style, brightness, and signal-to-noise ratio, eliminating resolution breaks and boundary discontinuities.

[0014] Balancing high-precision alignment in overlapping areas with semantic illusions in non-overlapping areas, pixel-level geometric correction and realistic high-frequency texture transfer can be achieved in overlapping areas, without ghosting, distortion, or misalignment artifacts; in non-overlapping areas, relying on multi-scale context and cross-attention, texture details with reasonable structure and consistent style can be generated, solving the problem of edge reconstruction collapse.

[0015] Achieving a smooth transition between two regions and visual consistency across the entire image is achieved through confidence-based feature-level adaptive fusion, resulting in a continuous gradient effect from overlapping to non-overlapping areas. This effect eliminates obvious boundaries and resolution breaks, ensuring global uniformity in clarity, color, and texture style, and creating a natural and coherent visual experience.

[0016] Achieving high-quality, high-consistency full-image super-resolution reconstruction maintains high-fidelity details and stable image quality across the entire field of view, breaking through the limitations of existing technologies where the center is clear but the edges are blurry, enabling the dual-camera zoom system to output complete and usable high-resolution images.

[0017] It boasts high computational efficiency and enhanced real-time performance on mobile devices. Feature extraction, matching, and fusion all support a lightweight architecture, with most computations performed in a low-resolution feature space. This results in low model parameter count and computational latency, meeting the real-time zoom and preview requirements of mobile phones.

[0018] It is more robust and applicable to a wider range of scenarios. It has stable reconstruction capabilities in situations with large parallax, complex deformation, occlusion, and changes in scene depth. It is suitable for both close-up portraits and distant scenes with large field of view such as buildings and landscapes.

[0019] The image signal-to-noise ratio and detail fidelity are significantly improved. The overlapping area uses real reference texture, and the non-overlapping area uses semantic-driven generation. Compared with simple interpolation or single-image super-resolution, the noise is lower, the edges are sharper, and the textures are more natural. Both subjective image quality and objective indicators are significantly improved.

[0020] Effectively expanding the effective field of view of dual-camera imaging transforms traditionally unusable edge blind spots into high-quality reconstructed areas, enhancing shooting composition flexibility and allowing for clear images across the entire field of view without post-processing cropping.

[0021] It has strong generalization ability and better adaptability to different scenarios. Multi-scale features and global attention mechanism can learn general visual priors. It can stably output high-quality results for different content such as repetitive textures, smooth regions, and complex structures, and is not prone to abnormal textures and distortions.

[0022] Furthermore, in S2, when the feature extraction module adopts an architecture based on a residual convolutional network, the residual convolutional network consists of multiple cascaded residual blocks, each of which contains a convolutional layer, a ReLU activation layer, and skip connections; when adopting an architecture based on a visual Transformer, it is a Swing Transformer module that includes a window moving attention mechanism; when adopting a lightweight network architecture, it is constructed from depthwise separable convolutional or channel shuffling units, reducing the number of model parameters and computational latency while ensuring feature representation capabilities.

[0023] Furthermore, the formula for calculating the cosine similarity matrix mentioned in S3 is as follows: The formulas for generating the confidence plot and index plot are as follows: , ;in, , They are respectively , The expanded local feature block vectors at the i and j-th positions. The L2 norm of a vector. Represents vector dot product operation; pre-distorted feature map The formula for generating it is: , where p is the spatial coordinate on the feature map, and P(p) is the best reference matching index corresponding to that position.

[0024] Furthermore, the deformable alignment module described in S4-1, when employing a feature alignment strategy based on deformable convolution, utilizes a lightweight alignment network to predict the offset Δp of each feature point in the x and y directions, and combines this with deformable convolutional layers. Resampling is performed; when using an optical flow-based alignment strategy, calculation is performed through an optical flow estimation network. and The dense optical flow field between them is used to perform spatial transformation on the reference features to achieve pixel-level alignment; when adopting a global alignment strategy based on affine or homography transformation, the network predicts a global 3×3 transformation matrix, and the matrix is ​​used to perform overall correction on the reference feature map.

[0025] Furthermore, in S4-2... The method for performing multi-scale context aggregation is as follows: using average pooling layers with kernel sizes of 3×3, 5×5, and 7×7 and a stride of 1. Feature extraction can be performed, or dilated convolutional and pyramid pooling modules with different dilation rates can be applied. The process involves combining the obtained multi-scale features with the original... Generate by element-wise addition The formula is: ,in This represents the average pooling operation with a kernel size of k.

[0026] Furthermore, the implementation formula for the cross-attention mechanism described in S4-2 is as follows: , , , Non-overlapping region mask The construction formula is Enhanced feature maps The formula for generating it is: ;in, , , Let be the learnable projection matrix, d be the number of feature channels, and τ be the threshold. The output features of the attention map.

[0027] Furthermore, in the adaptive dual-stream fusion module described in S5, when employing a confidence-based linear weighted fusion strategy, the fusion formula is as follows: c represents the pixel value of the matching confidence map C; when using a channel-based convolutional fusion strategy, , The combined feature tensor is obtained by concatenating the matching confidence map C along the channel dimension. This tensor is then input into a fusion sub-network consisting of several layers of 1x1 and 3x3 convolutions to achieve adaptive fusion. When a dynamic selection fusion strategy based on a gating mechanism is adopted, a lightweight gating network generates a dynamic gating map G with values ​​ranging from (0,1). The fusion formula is as follows: .

[0028] Furthermore, the reconstruction module described in S6, when employing an efficient reconstruction strategy based on sub-pixel convolution, will... The input is decoded by a cascaded 3x3 convolutional layer, which expands the number of feature channels to [number missing]. Then, subpixel convolution rearranges the channel-dimensional data to the spatial dimension, outputting a high-resolution image, where r is the magnification factor. The number of output image channels; when using a stepwise reconstruction strategy based on deconvolution, the number of channels will be... The input is progressively upsampled through cascaded deconvolutional layers, with each layer coupled with an activation function and detailed corrections made through convolutional layers after upsampling. When employing a reconstruction strategy based on interpolation and post-processing, bilinear or bicubic interpolation is used to... The image is then magnified to the target resolution and input into a multi-layer convolutional reconstruction network for sharpening and artifact removal. Attached Figure Description

[0029] Figure 1 The diagram shows the logic block diagram of the super-resolution zoom reconstruction method based on dual self-cooperation. Figure 2 The logical block diagram for S3 feature matching and region discrimination; Figure 3 A logical block diagram for feature alignment of the overlapping region in S4-1; Figure 4 A logical block diagram for feature enhancement of the non-overlapping region in S4-2; Figure 5 The logical block diagram for S5 dual-stream feature fusion; Figure 6 A comparison of the reconstruction effects of the method of this invention and existing technologies in a real-world scenario. Detailed Implementation

[0030] The following detailed description illustrates the specific implementation method: Super-resolution zoom reconstruction methods based on dual self-cooperative methods (such as...) Figure 1 (As shown), including the following steps: S1 Image Acquisition: Captures low-resolution images to be reconstructed using a wide-angle lens. Capture high-resolution reference images of the same scene using a telephoto lens. ; S2 Feature Extraction: and The input feature extraction module extracts a low-resolution feature map containing both low-level local features and high-level semantic features. and reference feature map The feature extraction module is implemented using a residual convolutional network, a visual Transformer, or a lightweight network architecture. S3 Feature Matching and Region Determination (e.g.) Figure 2 As shown): and The input feature matching module uses sliding window technology to expand the two into a sequence of local feature blocks and performs L2 normalization. It calculates the cosine similarity matrix through matrix multiplication, and then performs maximum search and parameter search in the reference feature dimension to generate a matching confidence map C and an index map P. The matching confidence map C accurately distinguishes the overlapping and non-overlapping regions of the image to be reconstructed, and the index map P records the spatial coordinates of the most similar texture in the reference feature map. S4 dual-branch feature reconstruction: S4-1 Overlapping region feature alignment (such as...) Figure 3 As shown): Based on the index graph P pair Pre-distorted feature maps are obtained by performing block-based hard distortion. Then, through a learnable and deformable alignment module... Fine alignment is performed to obtain the alignment feature map. The deformable alignment module is implemented using a global alignment strategy based on deformable convolution, optical flow estimation, or affine / homography transformation. S4-2 Feature enhancement of non-overlapping regions (e.g.) Figure 4 As shown): For Perform multi-scale context aggregation to obtain a multi-scale reference feature library. ,Will Mapped to query vector Q, Mapped to key vector K and value vector V respectively, attention map A is generated through cross-attention mechanism, and non-overlapping region mask is constructed by combining it with matching confidence map C. After filtering the output of attention map A, it is compared with... Weighted recombination yields enhanced feature maps of non-overlapping regions. ; S5 dual-stream feature fusion (such as...) Figure 5 As shown): and The input is processed by an adaptive dual-stream fusion module, which employs a dynamic selection fusion strategy based on confidence-based linear weighting, channel convolutional fusion, or gating mechanisms to perform feature-level adaptive fusion, resulting in a full-field fused feature map. This enables a smooth transition between overlapping and non-overlapping regions. S6 Super-Resolution Reconstruction: The input reconstruction module employs upsampling strategies combining subpixel convolution, deconvolution, or interpolation with post-processing to map the low-resolution feature space back to the high-resolution image space, outputting a full-field high-resolution super-resolution reconstructed image.

[0031] Specifically, in the target scene, the dual-focal lens captures low-resolution images. and high-resolution images These two images were captured from different fields of view and focal lengths by a short focal length lens and a long focal length lens, respectively; the low-resolution image... Used for subsequent image zoom processing, while the reference image Provides high-resolution detail information. Specific low-resolution images. and reference image The input feature extraction module uses a convolutional neural network (CNN) to extract both low-level and high-level features of the image. The CNN consists of multiple convolutional layers and residual blocks, with each layer extracting features at different levels. Low-level features include local information such as edges and corners, while high-level features contain the image's global structure and semantic information. After feature extraction, a low-resolution image is obtained. and reference image The feature map, i.e. and These represent low-resolution images. and reference image The extracted feature map contains multiple levels of low-level and high-level features.

[0032] The feature extraction module can be implemented using various network architectures. In this embodiment, a residual convolutional network-based extraction method is used. The feature extraction module consists of multiple cascaded residual blocks, each containing a convolutional layer, a ReLU activation layer, and skip connections. This structure effectively alleviates the gradient vanishing problem and extracts deep features containing rich information such as edges and corners.

[0033] In another embodiment, visual Transformer-based extraction employs a Transformer architecture based on a self-attention mechanism (such as the Swing Transformer module) to capture long-range global dependencies. Utilizing a window-moving attention mechanism instead of traditional convolution can extract features with stronger semantic representation capabilities and a wider receptive field, making it particularly suitable for feature matching in scenarios with large parallax.

[0034] In another embodiment, considering the computing power limitations of mobile terminals, lightweight feature extraction is based on lightweight networks. A lightweight feature extraction module is constructed using depthwise separable convolutional or channel shuffling units. While maintaining feature representation capabilities, this significantly reduces the number of model parameters and computational latency, meeting the requirements of real-time zoom.

[0035] Extracted low-resolution image features Features of the reference image The input feature matching module performs block-based dense matching within the feature space. It generates a matching confidence map and an index map by calculating the cosine similarity matrix, thereby accurately distinguishing overlapping and non-overlapping regions in the scene. Specifically, it first uses a sliding window technique to segment the low-resolution feature map... and reference feature map The sequence is expanded into local feature blocks, and L2 normalization is performed on each feature block vector. This normalization operation aims to eliminate the interference of feature magnitude on similarity calculation, thereby allowing the matching process to focus on the directional consistency of the texture structure. Subsequently, matrix multiplication is used to calculate... Each query block in The dot product of all key blocks within the search range is used to generate a cosine similarity matrix that characterizes texture correlation. Finally, a maximum value search and a parameter search are performed on the reference feature dimension of the similarity matrix to obtain the best matching score and position index, respectively, thereby generating a matching confidence map C and an index map P. The similarity calculation method can be as follows: S: Subsequently, confidence plot C and index P are generated: ,in, and These represent the i-th and j-th local feature block vectors after being expanded in the low-resolution feature map and the reference feature map, respectively; Representing vectors Norm; This represents the vector dot product operation. ∈[-1, 1] represents the cosine of the angle between two feature blocks in the feature space. A higher value indicates a more reliable texture match at the current location, while It records the spatial coordinates of the most similar texture in the reference image.

[0036] In the reconstruction module of overlapping regions, a hierarchical alignment of reference features from coarse to fine is required. Through spatial warping and deformability correction, the high-frequency textures of the reference image are accurately transferred to the feature space of the low-resolution image. Because the index map obtained solely through block matching can only solve large-scale displacement problems and is prone to block artifacts, a cascaded alignment strategy is also designed: first, the index map P is used for block-based hard warping to achieve coarse alignment of the content, which is then used to reconstruct a coarsely aligned pre-warped feature map. This process is equivalent to performing a resampling operation in the reference feature space, which will be compared with the current... The reference texture with the most similar location is "transferred". This can be represented as: in, Represents the spatial coordinates on the feature map. This is the best reference matching index corresponding to that position. The result obtained at this point... Although in terms of content and While the alignment is close, discontinuities or minor geometric misalignments may still exist in the spatial structure. Therefore, a learnable, deformable alignment module is introduced to eliminate these misalignments through a fine-grained alignment network based on residuals. Spatial misalignment in the middle.

[0037] To achieve this precise alignment goal, this embodiment employs the following alignment strategy: feature alignment based on deformable convolution, using a lightweight alignment network to predict the offset Δp of each feature point in the x and y directions, and combining deformable convolutional layers to resample the pre-distorted features. This approach can flexibly handle non-rigid deformation and occlusion issues, achieving high alignment accuracy and fewer artifacts.

[0038] In another embodiment, the alignment strategy employed is as follows: alignment based on optical flow, specifically, using an optical flow estimation network to calculate... and The dense optical flow field between them. The calculated optical flow field is used to perform spatial transformation on the reference features, thereby achieving pixel-level alignment. The optical flow method has good physical interpretability in handling continuous motion and rigid transformation scenarios.

[0039] In another embodiment, the alignment strategy employed is as follows: Global alignment based on affine or homography transformations is used for simple scenes with small disparities or inconsistencies in scene depth. This strategy employs a global homography matrix-based alignment. The network only needs to predict a single global 3×3 transformation matrix to perform overall correction of the reference features. This approach has extremely low computational overhead and is suitable as an alignment scheme in low-power modes.

[0040] After this step Not only did it retain the high-frequency texture of the reference image, but it also matched the low-resolution image in terms of geometry. This maintained a high degree of consistency, providing high-quality input for subsequent feature fusion.

[0041] In the reconstruction module for non-overlapping regions, an enhancement mechanism based on multi-scale perceptual cross-attention is introduced. This is because in non-overlapping regions, reference images cannot provide spatially aligned pixel-level information, but often contain semantic information consistent with the texture style of the region to be reconstructed (such as the same material, repeating patterns, etc.). To fully utilize these non-aligned reference cues, this module chooses to construct a multi-scale reference feature library. First, the extracted reference features... Multi-scale context aggregation was performed using average pooling layers with kernel sizes of 3×3, 5×5 and 7×7 to extract texture feature distributions under different receptive fields.

[0042] It should be understood that the method of constructing multi-scale context is not limited to average pooling. In other embodiments, dilated convolutions with different dilation rates or pyramid pooling modules can also be used to obtain multi-scale context information, all of which fall within the scope of protection of this invention. Subsequently, the features at these three scales are added element-wise to the original features to generate a reference feature library rich in multi-scale context information. This step ensures that the subsequently generated key-value vectors have stronger semantic expressive power, which can be represented as: in, Indicates the kernel size as The average pooling operation with a stride of 1 enriches the frequency domain information of the features by preserving their spatial resolution. Simultaneously, it integrates the previously generated multi-scale features... The vectors are projected onto the key vector K and the value vector V through two independent 1*1 convolutional layers, respectively.

[0043] To find the optimal texture match, calculate the dot product between Q and the transpose of K, and divide by the scaling factor. A normalized attention map is generated by smoothing the gradient and applying the Softmax function. This attention map describes the semantic relevance of each location in the low-resolution image to all locations in the reference image. It is represented as: , ,in, , , The projection matrix is ​​learnable. The number of feature channels is used. Then, a binary mask for the non-overlapping region is constructed using the generated confidence map C. To prevent the "phantom" textures generated by cross-attention from interfering with high-confidence overlapping regions, a mask is used. Attention output Empty filtering is performed. Finally, the filtered enhanced features are weighted and recombined with the original low-resolution features to output the enhanced features of non-overlapping regions. , is represented as: , , The result obtained through this step Focus on semantic detail recovery of non-overlapping regions.

[0044] In the adaptive dual-stream fusion module, the core task is to organically combine the alignment features of overlapping regions with the enhancement features of non-overlapping regions. To ensure smooth, seamless stitching in the field-of-view transition region, the following scheme is adopted in this embodiment: Linear weighted fusion based on confidence is employed, using a linear weighting strategy for feature fusion. Specifically, the previously generated matching confidence map C is used as spatial attention weights, and the alignment features and enhancement features are summed pixel-by-pixel with weights. The fusion formula can be expressed as follows: In this implementation, the confidence map C acts as a "soft switch": in overlapping regions with high texture matching, C approaches 1, and the model primarily adopts geometrically corrected aligned features; while in non-overlapping regions with low texture matching, C approaches 0, and the model automatically and smoothly transitions to enhanced features generated using cross-attention. This approach has low computational complexity and clear physical meaning, effectively avoiding stitching artifacts.

[0045] In another embodiment, the following approach is adopted: channel-based convolutional fusion. To capture the more complex nonlinear relationships between the two feature paths, a channel-based convolutional fusion strategy is employed. First, the aligned features, enhanced features, and confidence map C are concatenated along the channel dimension to construct a combined feature tensor with an increased number of channels. Subsequently, this combined feature is input into a fusion sub-network consisting of several layers of 1*1 and 3*3 convolutions. Through end-to-end training, this fusion sub-network can adaptively learn the optimal combination of the two feature paths, automatically correcting potential estimation biases in the confidence map, thereby achieving a more robust fusion effect in edge regions.

[0046] In another embodiment, the following scheme is adopted: dynamic selection fusion based on gating mechanism, introducing a learnable gating mechanism to control the information flow. Specifically, a lightweight gating network is designed, which receives alignment features and enhancement features as input and outputs a dynamically gated graph G with values ​​ranging from (0, 1). The final fusion feature is calculated as follows: · +(1-G)· Unlike directly using a fixed confidence map C, the gated map G is dynamically generated by the network based on the semantic consistency of the feature content. It not only focuses on the similarity of textures but also on the semantic continuity of features, thus providing greater flexibility in dealing with occlusion problems caused by large parallax.

[0047] In the final reconstruction module, the core task is to map the preceding full-field fusion features back from the low-resolution feature space to the high-resolution image space, restoring the image's details and texture. In this embodiment, the following scheme is adopted: Based on efficient reconstruction using sub-pixel convolution, this invention employs sub-pixel convolutional layers as the core upsampling operator. Specifically, the fusion features are first input into a set of cascaded 3x3 convolutional layers for deep feature decoding and texture restoration; subsequently, the number of feature channels is expanded to [a higher level] through a convolutional layer. (where r is the magnification factor, To output the image with 3 channels, subpixel convolution is used to periodically rearrange the channel-dimensional data to the spatial dimension, directly outputting a high-resolution image. This method performs most of the computation in the low-resolution space, resulting in extremely high computational efficiency and making it suitable for mobile deployments.

[0048] In another embodiment, the following approach is adopted: based on deconvolutional progressive reconstruction, deconvolutional layers are used to upsample the features. In this approach, the fused features pass through several deconvolutional layers. Each deconvolutional layer is responsible for enlarging the spatial size of the feature map by a certain factor, while simultaneously performing non-linear mapping using an activation function (such as ReLU or PReLU). In high-magnification zoom scenarios, multiple deconvolutional layers can be cascaded to progressively enlarge the feature map. After each enlargement stage, a regular convolutional layer is used for detail correction, and finally, a reconstruction layer outputs a super-resolution image.

[0049] In another embodiment, the following approach is adopted: Reconstruction based on interpolation and post-processing, as a simple alternative implementation, employs a "interpolate first, refine later" strategy. Specifically, bilinear or bicubic interpolation operators are first used to directly enlarge the spatial size of the fused features to the target resolution. Subsequently, the enlarged feature map is input into a reconstruction network composed of multiple convolutional layers for post-processing. The advantage of this structure is that the upsampling process does not contain learnable parameters; the network focuses on sharpening blurred texture edges and removing artifacts in high-resolution space, and the structure is simple and easy to implement.

[0050] This embodiment proposes a dual-stream architecture with parallel processing of a "reference branch" and a "cross branch," cleverly resolving the fundamental contradiction of inconsistent field of view in dual-camera systems. Existing technologies often attempt to process the entire image using a single logic, making it difficult to simultaneously achieve alignment accuracy and generation capability. This invention overcomes this limitation by intelligently decoupling the full-image features through a confidence map: the reference branch focuses on precise geometric correction of overlapping regions, using deformable convolution to eliminate ghosting caused by large parallax; the cross branch focuses on texture mapping of non-overlapping regions, using multi-scale cross attention to search for global semantics from the reference map. This divide-and-conquer, collaborative dual-branch design is the core of achieving high-quality zoom reconstruction.

[0051] This embodiment achieves highly consistent full-image reconstruction across the entire field of view, eliminating the fragmented feel of traditional stitching schemes. Unlike the traditional "center cropping + edge filling" stitching process, this invention performs end-to-end full-image reconstruction in a deep feature space. Through a confidence-based adaptive fusion mechanism, the features extracted by the two branches are seamlessly combined at the sub-pixel level. This ensures that the final high-resolution output image maintains high global consistency in brightness, color reproduction, texture style, and signal-to-noise ratio, presenting a natural and continuous visual effect for both the central subject and the edge background.

[0052] This embodiment effectively solves the "field of view blind spot" problem in non-overlapping areas of dual-camera zoom, significantly improving the usability of image edges. Addressing the inherent non-overlapping field of view between wide-angle and telephoto lenses, this invention utilizes the global search capability of cross-branching to successfully "migrate" texture details from the reference image to the field of view blind spot. This not only avoids edge blurring and stretching distortion caused by traditional interpolation magnification but also results in a reconstructed full image with rich high-frequency details, greatly expanding the effective field of view and compositional flexibility of the dual-camera zoom system in actual shooting.

[0053] To further illustrate the effectiveness of this embodiment, existing methods such as LR, TTSR, and GT were selected for comparison. LR refers to the original low-resolution image to be reconstructed; in this comparative experiment, it specifically refers to the image upsampled to the target resolution using a bicubic interpolation algorithm, used to measure the basic imaging. TTSR is a classic example of reference image super-resolution. It constructs a texture Transformer architecture and uses an attention mechanism to search for and transfer texture features in the reference image. TTSR performs well in overlapping regions, but when dealing with non-overlapping regions (edge ​​blind spots) caused by asymmetric dual cameras, it often introduces incorrect reference information due to a lack of effective region discrimination mechanisms, leading to artifacts. GT, as the original high-resolution raw data captured under the same optical axis, contains the most realistic and rich sub-pixel level details and structural features of the scene, serving as a high-resolution ground truth image of the target scene. By performing a depth comparison between the reconstructed image of this invention and the GT, it is possible to... Figure 6 The process verifies whether the details generated by the dual self-cooperative mechanism proposed in this invention in the semantic illusion of non-overlapping regions conform to the physical reality.

[0054] Reconstruction results as follows Figure 6 As shown, from Figure 6 The following conclusions can be drawn: the proposed dual-self-cooperative super-resolution reconstruction method significantly outperforms other comparative methods across the entire field of view. In image detail restoration, LR, due to its simple upsampling, suffers from severe blurring and jagged edges in high-frequency areas such as building railings, wall brick textures, and motorcycle wheel hubs. While TTSR attempts to utilize the texture of the reference image, its reconstruction results show significant texture distortion in the brick texture area and structural distortion and detail loss in the motorcycle wheel hub area when facing geometric deformation and non-overlapping regions caused by dual-camera parallax, failing to accurately restore the true geometric features of the object. In contrast, the proposed solution, through a "dual-self-cooperative" mechanism, achieves pixel-level precise texture transfer in overlapping areas using a deformable alignment module, resulting in clear, sharp, and structurally complete details such as railing edges and wheel hub spokes. In non-overlapping areas, the proposed method utilizes a multi-scale cross-attention mechanism for semantically reasonable reconstruction, avoiding artifacts caused by blind alignment. The generated wall brick texture is highly consistent with the surrounding area in terms of distribution logic and visual style. In summary, the image reconstructed by the present invention is closest to the ground truth in terms of visual clarity, structural fidelity, and overall texture consistency. It effectively overcomes the problem of image collapse in edge regions in existing technologies and achieves high-quality full-image super-resolution reconstruction.

[0055] The above are merely embodiments of the present invention. The invention is not limited to the fields covered by these embodiments. Commonly known structures and characteristics in the solutions are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are able to access all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A super-resolution zoom reconstruction method based on dual self-cooperation, characterized in that, Includes the following steps: S1 Image Acquisition: Captures low-resolution images to be reconstructed using a wide-angle lens. Capture high-resolution reference images of the same scene using a telephoto lens. ; S2 Feature Extraction: and The input feature extraction module extracts a low-resolution feature map containing both low-level local features and high-level semantic features. and reference feature map The feature extraction module is implemented using a residual convolutional network, a visual Transformer, or a lightweight network architecture. S3 Feature Matching and Region Determination: [The sentence is incomplete and likely refers to a specific feature matching and region discrimination method. and The input feature matching module uses sliding window technology to expand the two into a sequence of local feature blocks and performs L2 normalization. It calculates the cosine similarity matrix through matrix multiplication, and then performs maximum search and parameter search in the reference feature dimension to generate a matching confidence map C and an index map P. The matching confidence map C accurately distinguishes the overlapping and non-overlapping regions of the image to be reconstructed, and the index map P records the spatial coordinates of the most similar texture in the reference feature map. S4 dual-branch feature reconstruction: S4-1 Overlapping Region Feature Alignment: Based on Index Graph P Pre-distorted feature maps are obtained by performing block-based hard distortion. Then, through a learnable and deformable alignment module... Fine alignment is performed to obtain the alignment feature map. The deformable alignment module is implemented using a global alignment strategy based on deformable convolution, optical flow estimation, or affine / homography transformation. S4-2 Non-overlapping region feature enhancement: [The text abruptly ends here, likely due to an incomplete sentence Perform multi-scale context aggregation to obtain a multi-scale reference feature library. ,Will Mapped to query vector Q, Mapped to key vector K and value vector V respectively, attention map A is generated through cross-attention mechanism, and non-overlapping region mask is constructed by combining it with matching confidence map C. After filtering the output of attention map A, it is compared with... Weighted recombination yields enhanced feature maps of non-overlapping regions. ; S5 Dual-Stream Feature Fusion: and The input is processed by an adaptive dual-stream fusion module, which employs a dynamic selection fusion strategy based on confidence-based linear weighting, channel convolutional fusion, or gating mechanisms to perform feature-level adaptive fusion, resulting in a full-field fused feature map. This achieves a smooth transition between overlapping and non-overlapping areas; S6 Super-Resolution Reconstruction: The input reconstruction module employs upsampling strategies combining subpixel convolution, deconvolution, or interpolation with post-processing to map the low-resolution feature space back to the high-resolution image space, outputting a full-field high-resolution super-resolution reconstructed image.

2. The super-resolution zoom reconstruction method based on dual self-cooperation according to claim 1, characterized in that, The feature extraction module described in S2, when adopting a residual convolutional network-based architecture, consists of multiple cascaded residual blocks, each containing a convolutional layer, a ReLU activation layer, and skip connections; when adopting a visual Transformer-based architecture, it is a Swing Transformer module containing a window moving attention mechanism; when adopting a lightweight network architecture, it is constructed from depthwise separable convolutional or channel shuffling units, reducing the number of model parameters and computational latency while ensuring feature representation capabilities.

3. The super-resolution zoom reconstruction method based on dual self-cooperation according to claim 2, characterized in that, The formula for calculating the cosine similarity matrix mentioned in S3 is as follows: The formulas for generating the confidence plot and index plot are as follows: , ;in, , They are respectively , The expanded local feature block vectors at the i and j-th positions. Describes the L2 norm of a vector. Represents vector dot product operation; pre-distorted feature map The formula for generating it is: , where p is the spatial coordinate on the feature map, and P(p) is the best reference matching index corresponding to that position.

4. The super-resolution zoom reconstruction method based on dual self-cooperation according to claim 3, characterized in that, The deformable alignment module described in S4-1, when employing a feature alignment strategy based on deformable convolution, utilizes a lightweight alignment network to predict the offset Δp of each feature point in the x and y directions, and combines this with deformable convolutional layers. Resampling is performed; when using an optical flow-based alignment strategy, calculation is performed through an optical flow estimation network. and The dense optical flow field between them is used to perform spatial transformation on the reference features to achieve pixel-level alignment; when adopting a global alignment strategy based on affine or homography transformation, the network predicts a global 3×3 transformation matrix, and the matrix is ​​used to perform overall correction on the reference feature map.

5. The super-resolution zoom reconstruction method based on dual self-cooperation according to claim 4, characterized in that, S4-2 The method for performing multi-scale context aggregation is as follows: using average pooling layers with kernel sizes of 3×3, 5×5, and 7×7 and a stride of 1. Feature extraction can be performed, or dilated convolutional and pyramid pooling modules with different dilation rates can be applied. The process involves combining the obtained multi-scale features with the original... Generate by element-wise addition The formula is: ,in This represents the average pooling operation with a kernel size of k.

6. The super-resolution zoom reconstruction method based on dual self-cooperation according to claim 5, characterized in that, The implementation formula for the cross-attention mechanism described in S4-2 is as follows: , , , Non-overlapping region mask The construction formula is Enhanced feature maps The formula for generating it is: ;in, , , Let be the learnable projection matrix, d be the number of feature channels, and τ be the threshold. The output features of the attention map.

7. The super-resolution zoom reconstruction method based on dual self-cooperation according to claim 6, characterized in that, The adaptive dual-stream fusion module described in S5, when employing a confidence-based linear weighted fusion strategy, uses the following fusion formula: c is the pixel value of the matching confidence map C; When employing a channel-based convolutional fusion strategy, , The combined feature tensor is obtained by concatenating the matching confidence map C along the channel dimension. This tensor is then input into a fusion sub-network consisting of several layers of 1x1 and 3x3 convolutions to achieve adaptive fusion. When a dynamic selection fusion strategy based on a gating mechanism is adopted, a lightweight gating network generates a dynamic gating map G with values ​​ranging from (0,1). The fusion formula is as follows: .

8. The super-resolution zoom reconstruction method based on dual self-cooperation according to claim 7, characterized in that, The reconstruction module described in S6, when employing an efficient reconstruction strategy based on sub-pixel convolution, will... The input is decoded by a cascaded 3x3 convolutional layer, which expands the number of feature channels to [number missing]. Then, subpixel convolution rearranges the channel-dimensional data to the spatial dimension, outputting a high-resolution image, where r is the magnification factor. The number of output image channels; when using a stepwise reconstruction strategy based on deconvolution, the number of channels will be... The input is progressively upsampled through cascaded deconvolutional layers, with each layer coupled with an activation function and detailed corrections made through convolutional layers after upsampling. When employing a reconstruction strategy based on interpolation and post-processing, bilinear or bicubic interpolation is used to... The image is then magnified to the target resolution and input into a multi-layer convolutional reconstruction network for sharpening and artifact removal.

9. A super-resolution zoom reconstruction system based on dual self-cooperation, characterized in that, The method described in any one of claims 1-8 was adopted.

10. The super-resolution zoom reconstruction system based on dual self-cooperation according to claim 9, characterized in that, include: The image acquisition module is used to capture low-resolution images to be reconstructed using a wide-angle lens. Capture high-resolution reference images of the same scene using a telephoto lens. ; The feature extraction module, employing residual convolutional networks, visual Transformers, or lightweight network architectures, is used to receive... and Low-resolution feature maps containing low-level local features and high-level semantic features are extracted. and reference feature map ; Feature matching module, used to receive and After feature block expansion, L2 normalization, and cosine similarity matrix calculation, a matching confidence map C and an index map P are generated to distinguish the overlapping and non-overlapping regions of the image to be reconstructed. The dual-branch reconstruction module includes an overlapping region alignment unit and a non-overlapping region enhancement unit. The overlapping region alignment unit incorporates a deformable alignment module employing deformable convolution, optical flow estimation, or affine / homophonic transformation strategies for... Alignment feature maps are obtained by performing coarse and fine alignment. The non-overlapping region enhancement unit is used for... Construct a multi-scale reference feature library, and generate enhanced feature maps through cross-attention mechanism and mask filtering. ; Adaptive dual-stream fusion module, used for receiving and Feature-level fusion is performed using a dynamic selection fusion strategy that employs confidence-based linear weighting, channel convolutional fusion, or gating mechanisms to obtain a full-field-of-view fused feature map. ; Image reconstruction module, used to receive It employs upsampling strategies combining subpixel convolution, deconvolution, or interpolation with post-processing to map the low-resolution feature space back to the high-resolution image space, outputting a full-field high-resolution super-resolution reconstructed image.

Images