Image super-resolution reconstruction method and device based on any magnification of double cameras

Abstract

The invention relates to an arbitrary magnification image super-resolution reconstruction method and device based on double cameras. The method comprises the following steps: acquiring a to-be-processed image and a reference image; determining the scale of the target resolution image based on the target magnification, and generating a target resolution coordinate set corresponding to the scale; and on the basis of the target resolution coordinate set, a preset super-resolution model is utilized to perform query operation and reconstruction operation on the to-be-processed image and the reference image to generate a target resolution image, and the preset super-resolution model comprises a space reference space, a luminosity reference space and a texture reference space. Therefore, the mixed reference space model containing the space reference space, the luminosity reference space and the texture reference space is constructed and utilized to carry out parallel query and reconstruction on the image features in different levels of reference spaces; the problems of poor generalization ability and luminosity distortion or texture blurring of the reconstructed image caused by fixed amplification factor and single alignment space in the prior art are solved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and machine learning technology, and in particular to a method and apparatus for super-resolution reconstruction of images at arbitrary magnification using dual cameras. Background Technology

[0002] Single Image Super-resolution (SISR) technology aims to reconstruct a high-resolution image from a single low-resolution image, and it has wide applications in fields such as mobile imaging, medical imaging, satellite remote sensing, and security monitoring. However, SISR has an inherent ill-posed inverse problem: the same low-resolution image may correspond to countless different high-resolution images. This leads to serious uncertainties in the reconstruction process, making it difficult to recover true and clear details.

[0003] In related technologies, a typical dual-camera super-resolution scheme can be summarized as follows: First, a low-resolution image to be processed and a high-resolution reference image are acquired simultaneously through a dual-camera system; second, features of both images are extracted using a neural network, and feature matching and alignment are performed in a single feature space; finally, the aligned features are reconstructed through an upsampling network designed for a fixed magnification to obtain a high-resolution image.

[0004] However, this method is designed and trained only for specific, discrete magnifications, and cannot flexibly respond to users' needs for continuous zoom or super-resolution at any specified scale. Furthermore, this method only performs coarse feature matching in a single spatial domain or feature domain, resulting in photometric distortion or texture blurring in the reconstructed image, which urgently needs to be addressed. Summary of the Invention

[0005] This application provides a method and apparatus for super-resolution reconstruction of images based on dual cameras at arbitrary magnification, in order to solve the problems of poor generalization ability and optical distortion or texture blurring in the reconstructed images caused by fixed magnification and single alignment space in related technologies. It achieves the technical effect of high-quality, high-fidelity super-resolution reconstruction of low-resolution images of real dual-camera systems at any user-specified magnification.

[0006] To achieve the above objectives, the first aspect of this application proposes an image super-resolution reconstruction method, comprising the following steps: Obtain an image to be processed and a reference image, wherein the resolution of the image to be processed is lower than the resolution of the reference image; Based on the target magnification, the scale of the target resolution image is determined, and a set of target resolution coordinates corresponding to the scale is generated; Based on the target resolution coordinate set, a preset super-resolution model is used to perform query and reconstruction operations on the image to be processed and the reference image to generate the target resolution image. The preset super-resolution model includes a null reference space, a photometric reference space, and a texture reference space.

[0007] According to one embodiment of this application, the step of generating the target resolution image by performing query and reconstruction operations on the image to be processed and the reference image based on the target resolution coordinate set and using a preset super-resolution model includes: The feature extraction network of the preset super-resolution model is used to extract the source image features of the image to be processed and the sampled reference features of the reference image, wherein the sampled reference features include downsampled reference features and upsampled reference features; The target resolution coordinate set is divided into first to third subsets according to a preset ratio. Based on the source image features, the coordinates corresponding to the first subset are queried using the empty reference space to obtain the first set of target resolution pixel values. Based on the source image features and the downsampling reference features, the coordinates corresponding to the second subset are queried using the photometric reference space to obtain the second set of target resolution pixel values. Based on the upsampling features, the coordinates corresponding to the third subset are queried using the texture reference features to obtain the third set of target resolution pixel values. The target resolution image is obtained by fusing the first set of target resolution pixel values, the second set of target resolution pixel values, and the third set of target resolution pixel values.

[0008] According to one embodiment of this application, the step of querying the coordinates corresponding to the second subset using the photometric reference space based on the source image features and the downsampled reference features to obtain the second set of target resolution pixel values ​​includes: The source image features and the downsampled reference features are photometrically aligned using the photometric reference space to generate photometric aligned features. Based on the photometric alignment feature, a query operation is performed on the coordinates corresponding to the second subset to obtain the second set of target resolution pixel values.

[0009] According to one embodiment of this application, the step of querying the coordinates corresponding to the third subset based on the upsampling features and using the texture reference features to obtain the third set of target resolution pixel values ​​includes: The photometric alignment feature and the upsampling feature are texture aligned using the texture reference space to generate a texture alignment feature. Based on the texture alignment features, a query operation is performed on the coordinates corresponding to the third subset to obtain the third set of target resolution pixel values.

[0010] According to one embodiment of this application, the step of performing photometric alignment processing on the source image features and the downsampled reference features using the photometric reference space to generate photometric aligned features includes: The source image features and the downsampled reference features are fused to obtain the downsampled fused features; Spatial pooling operations are performed on the texture alignment features and the downsampled reference features respectively to obtain the first photometric representation of the image to be processed and the second photometric representation of the reference image; Based on the first photometric characterization and the second photometric characterization, the downsampling fusion feature is modulated to obtain the photometric alignment feature.

[0011] According to one embodiment of this application, the step of performing texture alignment processing on the photometric alignment feature and the upsampling feature using the texture reference space to generate texture alignment features includes: The photometric alignment feature and the upsampling feature are input into a preset dilated convolutional neural network to generate an upsampling index map and an upsampling confidence map. Based on the upsampled index map, target feature blocks are extracted from the reference image; Based on the upsampled confidence map, the target feature block is weighted and fused to obtain the texture alignment feature.

[0012] According to an embodiment of this application, a method for super-resolution reconstruction of images based on arbitrary magnification of dual-camera systems is proposed. This method determines the scale of the target resolution image based on the target magnification and generates a target resolution coordinate set corresponding to the scale. Based on the target resolution coordinate set, a preset super-resolution model is used to perform query and reconstruction operations on the image to be processed and a reference image to generate the target resolution image. The preset super-resolution model includes a null reference space, a photometric reference space, and a texture reference space. Therefore, by constructing and utilizing a hybrid reference space model containing null, photometric, and texture reference spaces, image features are queried and reconstructed in parallel across different levels of reference spaces. This solves the problems of poor generalization ability and photometric distortion or texture blurring in reconstructed images caused by fixed magnification and a single alignment space in related technologies. It achieves high-quality, high-fidelity super-resolution reconstruction of low-resolution images from real dual-camera systems at any user-specified magnification.

[0013] To achieve the above objectives, a second aspect of this application provides a device for super-resolution reconstruction of images at arbitrary magnification using dual cameras, comprising: An acquisition module is used to acquire an image to be processed and a reference image, wherein the resolution of the image to be processed is lower than the resolution of the reference image; The first generation module is used to determine the scale of the target resolution image based on the target magnification and generate a set of target resolution coordinates corresponding to the scale. The second generation module is used to perform query and reconstruction operations on the image to be processed and the reference image based on the target resolution coordinate set and using a preset super-resolution model to generate the target resolution image. The preset super-resolution model includes a null reference space, a photometric reference space and a texture reference space.

[0014] According to one embodiment of this application, the second generation module includes: The extraction unit is used to extract the source image features of the image to be processed and the sampled reference features of the reference image using the feature extraction network of the preset super-resolution model, wherein the sampled reference features include downsampled reference features and upsampled reference features; The first processing unit is used to divide the target resolution coordinate set into a first to a third subset according to a preset ratio, and to perform a query operation on the coordinates corresponding to the first subset based on the source image features and using the empty reference space to obtain a first set of target resolution pixel values. The second processing unit is used to perform a query operation on the coordinates corresponding to the second subset based on the source image features and the downsampling reference features, and to obtain the second set of target resolution pixel values ​​by using the photometric reference space. The third processing unit is used to perform a query operation on the coordinates corresponding to the third subset based on the upsampling features and the texture reference features to obtain the third set of target resolution pixel values. The fusion unit is used to fuse the first group of target resolution pixel values, the second group of target resolution pixel values ​​and the third group of target resolution pixel values ​​to obtain the target resolution image.

[0015] According to one embodiment of this application, the second processing unit includes: The first processing subunit is used to perform photometric alignment processing on the source image features and the downsampled reference features using the photometric reference space to generate photometric aligned features. The second processing subunit is used to perform a query operation on the coordinates corresponding to the second subset based on the photometric alignment feature, so as to obtain the second set of target resolution pixel values.

[0016] According to one embodiment of this application, the third processing unit is specifically used for: The third processing subunit is used to perform texture alignment processing on the photometric alignment feature and the upsampling feature using the texture reference space to generate texture alignment feature; The fourth processing subunit is used to perform a query operation on the coordinates corresponding to the third subset based on the texture alignment features to obtain the third set of target resolution pixel values.

[0017] According to one embodiment of this application, the first processing subunit is specifically used for: The source image features and the downsampled reference features are fused to obtain the downsampled fused features; Spatial pooling operations are performed on the texture alignment features and the downsampled reference features respectively to obtain the first photometric representation of the image to be processed and the second photometric representation of the reference image; Based on the first photometric characterization and the second photometric characterization, the downsampling fusion feature is modulated to obtain the photometric alignment feature.

[0018] According to one embodiment of this application, the third processing subunit is specifically used for: The photometric alignment feature and the upsampling feature are input into a preset dilated convolutional neural network to generate an upsampling index map and an upsampling confidence map. Based on the upsampled index map, target feature blocks are extracted from the reference image; Based on the upsampled confidence map, the target feature block is weighted and fused to obtain the texture alignment feature.

[0019] According to an embodiment of this application, a dual-camera arbitrary magnification image super-resolution reconstruction device is proposed. It determines the scale of the target resolution image based on the target magnification and generates a target resolution coordinate set corresponding to the scale. Based on the target resolution coordinate set, a preset super-resolution model is used to perform query and reconstruction operations on the image to be processed and a reference image to generate the target resolution image. The preset super-resolution model includes a null reference space, a photometric reference space, and a texture reference space. Therefore, by constructing and utilizing a hybrid reference space model containing null, photometric, and texture reference spaces, image features are queried and reconstructed in parallel across different levels of reference spaces. This solves the problems of poor generalization ability and photometric distortion or texture blurring in reconstructed images caused by fixed magnification and a single alignment space in related technologies. It achieves high-quality, high-fidelity super-resolution reconstruction of low-resolution images from real dual-camera systems at any user-specified magnification.

[0020] To achieve the above objectives, a third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement a dual-camera arbitrary magnification image super-resolution reconstruction method as described in the above embodiments.

[0021] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium storing a computer program that is executed by a processor to implement a dual-camera arbitrary magnification image super-resolution reconstruction method as described in the above embodiments.

[0022] To achieve the above objectives, a fifth aspect of this application provides a computer program product comprising a computer program that, when executed by a processor, is used to implement a dual-camera arbitrary magnification image super-resolution reconstruction method as described in the above embodiments.

[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a method for super-resolution reconstruction of images based on dual cameras at arbitrary magnification, according to an embodiment of this application. Figure 2 This is a schematic diagram of the architecture of a preset super-resolution model according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a photometric alignment component and a texture alignment component in a preset super-resolution model according to an embodiment of this application, wherein, Figure 3 (a) is a schematic diagram of the photometric alignment component. Figure 3 (b) is a schematic diagram of the texture alignment component; Figure 4 This is a schematic diagram illustrating the performance comparison experiment results according to an embodiment of this application; Figure 5 This is a block diagram illustrating a dual-camera arbitrary magnification image super-resolution reconstruction device according to an embodiment of this application. Figure 6 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0025] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0026] The following description, with reference to the accompanying drawings, describes a method and apparatus for super-resolution reconstruction of images at arbitrary magnification using dual cameras, according to embodiments of this application.

[0027] Figure 1 This is a flowchart of a method for super-resolution reconstruction of images at arbitrary magnification based on dual cameras, according to an embodiment of this application.

[0028] Before introducing the super-resolution reconstruction method for images at arbitrary magnification based on dual cameras proposed in the embodiments of this application, let's briefly introduce the relevant technical background.

[0029] With the development of single-image super-resolution technology, reference-based super-resolution (RefSR) methods have emerged. This method significantly improves the quality of reconstructed images by introducing an additional high-resolution reference image, providing prior information and detail guidance for the reconstruction process of low-resolution input. Among these, dual-camera super-resolution (SR), as an important application of reference-based super-resolution, utilizes the dual-camera systems commonly found in smartphones and other devices. It simultaneously acquires low-resolution and high-resolution reference images of the same scene using a telephoto lens and a wide-angle lens, respectively, demonstrating clear practical value and broad application prospects.

[0030] However, existing dual-camera super-resolution methods still face several significant limitations in practical applications: (1) Insufficient scale expansion capability: Most related technologies are designed for fixed magnification (such as ×2x), which cannot flexibly adapt to the super-resolution requirements of continuous zoom or user-defined scale, thus limiting their applicability in real-world scenarios.

[0031] (2) Mismatch between degradation model and real imaging: Related technologies rely on idealized degradation models such as bicubic downsampling for training, which makes it difficult to effectively model the complex imaging degradation introduced by lens differences, noise, compression and other factors in real dual-camera systems, resulting in a decrease in the generalization performance of the model on actual devices.

[0032] (3) Single feature alignment mechanism: Related technologies usually perform low-resolution alignment between the reference image and the reference image in a single feature space, which fails to fully explore the complementary information of the reference image at different semantic levels (such as overall illumination consistency and local texture structure), affecting the brightness naturalness and texture clarity of the reconstruction result.

[0033] Therefore, there is an urgent need for a dual-camera super-resolution method that can support arbitrary magnification scales, adapt to real imaging degradation processes, and perform efficient feature alignment in a multi-level reference space, so as to achieve high-quality, high-fidelity super-resolution reconstruction of low-resolution images from real dual-camera systems at any user-specified magnification.

[0034] For example, such as Figure 1 As shown, the super-resolution reconstruction method based on dual-camera images at arbitrary magnification includes the following steps: In step S101, an image to be processed and a reference image are obtained, wherein the resolution of the image to be processed is lower than the resolution of the reference image.

[0035] Understandably, the image to be processed refers to a low-resolution image captured by a telephoto lens, characterized by a narrow field of view and relatively low spatial resolution. It is the direct target for image quality enhancement and resolution improvement. The reference image refers to a high-resolution image of the same scene captured simultaneously by a wide-angle lens. It is characterized by a wide field of view, typically with higher original resolution and richer detail information, and is used to provide texture, structure, and luminosity priors for the super-resolution reconstruction of the image to be processed. Resolution is the number of pixels contained in an image, usually expressed as "width pixels × height pixels." The higher the resolution, the richer the details and the higher the sharpness of the image.

[0036] Specifically, the system first uses dual-camera hardware (such as the telephoto-wide-angle lens combination commonly found in smartphones) to simultaneously acquire two images of the same scene. The image captured by the telephoto lens is defined as the image to be processed. Due to optical design, imaging distance, or sensor configuration, its resolution is relatively low and lacks detail. The image captured by the wide-angle lens is defined as the reference image, which typically has a higher original resolution and more complete scene coverage. These two images are highly related in content, but differ in resolution, field of view, and level of detail. This constitutes a paired input relationship of "low-resolution target - high-resolution reference" in subsequent super-resolution processing, providing the necessary conditions for reference-guided feature alignment and reconstruction.

[0037] In step S102, the scale of the target resolution image is determined based on the target magnification, and a set of target resolution coordinates corresponding to the scale is generated.

[0038] Understandably, the target magnification ratio refers to the factor by which the user or system expects to enlarge the low-resolution image to the target image. It can be expressed as ×2, ×3, ×4, or any non-integer multiple (such as ×1.5). This parameter determines the extent to which the output image is spatially expanded relative to the input image. The scale of the target resolution image refers to the absolute pixel size of the output image calculated based on the target magnification ratio, which can be expressed as the number of pixels in width (W') and height (H'). For example, if the input image is 640×480 pixels and the target magnification ratio is ×3, then the scale of the target resolution image is 1920×1440 pixels. The target resolution coordinate set refers to a set of continuous or discrete position coordinates on the target scale image plane, which can be represented using normalized two-dimensional coordinates (x,y), where x,y∈[0,1] (or [-1,1]). Each coordinate corresponds to a pixel position in the target resolution image, and the target resolution coordinate set defines all positions where pixel values ​​need to be generated during the reconstruction process.

[0039] Specifically, the system first calculates the scale of the target resolution image to be generated based on the input target magnification, that is, determines the specific number of pixels in its width and height. Then, based on this scale information, a set of coordinates covering the entire target resolution image plane can be generated, where each coordinate point uniquely corresponds to a pixel position in the target resolution image.

[0040] In step S103, based on the target resolution coordinate set, a query operation and a reconstruction operation are performed on the image to be processed and the reference image using a preset super-resolution model to generate the target resolution image. The preset super-resolution model includes a null reference space, a photometric reference space and a texture reference space.

[0041] The query operation refers to taking coordinate points from the target resolution coordinate set as input and sending them to various reference spaces within the pre-defined super-resolution model (i.e., empty reference space, photometric reference space, and texture reference space, such as...). Figure 2 The process of initiating an "interrogation" (as shown) to obtain the feature response or reconstruction information of a given coordinate location in the corresponding space is called reconstruction. Reconstruction refers to the process of decoding the feature representation of a coordinate point in a reference space into its corresponding RGB (Red, Green, Blue) pixel values ​​using a dedicated mapping network for that space (such as a Multi-Layer Perceptron (MLP)). An empty reference space is used to extract low-resolution features; a photometric reference space is used to align photometric information; and a texture reference space is used to align texture information.

[0042] Specifically, after obtaining the target resolution coordinate set, this set can be used as a unified spatial index to perform query and reconstruction operations in parallel across the three structured reference spaces of the pre-defined super-resolution model. Ultimately, the pre-defined super-resolution model can fuse the pixels output from the three reference spaces according to their coordinates, thereby generating a complete target resolution image that maintains structural fidelity, photometric consistency, and clear texture. By performing multi-level alignment of the reference images in different reference spaces, the photometric consistency and texture detail of the reconstructed image are effectively improved.

[0043] Understandably, the target resolution coordinate set is essentially a "query request" to the pre-defined super-resolution model, specifying the spatial locations where pixel values ​​need to be generated. This mechanism frees the model from the limitations of traditional fixed upsampling structures, decoupling coordinate mapping at arbitrary scales from image features, and enabling flexible responses to users' arbitrary magnification needs, providing a scalable spatial indexing foundation for core reconstruction operations.

[0044] To make it easier to understand, the following details how to generate a target resolution image.

[0045] In some embodiments, as a possible implementation, a target resolution image is generated by performing query and reconstruction operations on the image to be processed and a reference image based on a target resolution coordinate set and a preset super-resolution model. This includes: extracting source image features of the image to be processed and sampled reference features of the reference image using a feature extraction network of the preset super-resolution model, wherein the sampled reference features include downsampling reference features and upsampling reference features; dividing the target resolution coordinate set into first to third subsets according to a preset ratio; querying the coordinates corresponding to the first subset using an empty reference space based on the source image features to obtain a first set of target resolution pixel values; querying the coordinates corresponding to the second subset using a photometric reference space based on the source image features and the downsampling reference features to obtain a second set of target resolution pixel values; querying the coordinates corresponding to the third subset using texture reference features based on the upsampling features to obtain a third set of target resolution pixel values; and fusing the first set of target resolution pixel values, the second set of target resolution pixel values, and the third set of target resolution pixel values ​​to obtain the target resolution image.

[0046] The feature extraction network refers to a backbone sub-network in the pre-defined super-resolution model, used to extract multi-scale, multi-level abstract feature representations from the input image, providing basic data for subsequent query operations in each reference space. A pre-trained convolutional neural network (such as VGG (Visual Geometry Group)) can be used as the feature extraction network. The pre-defined ratio can be fixed (e.g., 1:1:18) or dynamically adjusted according to the image content or training phase, used to control the contribution weights of the three reference spaces in the overall reconstruction.

[0047] Specifically, the feature extraction network of the pre-defined super-resolution model can extract the features of the image to be processed, I. LR Source image features F LR (i.e., low-resolution features), and reference image I Ref Sampling reference features at two scales, i.e., downsampling reference features F Ref 1× (Feature representation with spatial resolution reduced to the same scale as the image to be processed) and upsampled reference features F Ref 2× (The spatial resolution is upscaled to the same scale as the target resolution image's feature representation.) Subsequently, the target resolution coordinate set can be divided into three subsets (i.e., the empty coordinate set, the photometric coordinate set, and the texture coordinate set) according to a preset ratio. These three subsets can be processed in different reference spaces: the empty reference space can be based on the source image features F... LR The first multilayer perceptron is used to query the first subset (i.e., the empty coordinate set) to obtain pixels that retain their original structure (i.e., the first set of target resolution pixel values, empty high-resolution RGB pixel values); the photometric reference space can be fused with source image features F. LR With downsampled reference feature F Ref 1× The second multilayer perceptron is used to perform a photometric alignment query on the second subset (i.e., the photometric coordinate set) to obtain photometrically coordinated pixels (i.e., the second set of target resolution pixel values, photometric high-resolution RGB pixel values); the texture reference space can then be further fused with upsampled reference features F. Ref 2× The third subset (i.e., the texture feature set) is subjected to texture refinement query to obtain pixels with enhanced details (i.e., the third set of target resolution pixel values, texture high-resolution RGB pixel values). Finally, the preset super-resolution model can merge and fuse the pixel groups output from the three reference spaces according to their corresponding coordinate positions, thereby forming a target resolution image with complete structure, consistent color, and rich details.

[0048] Optionally, in some embodiments, based on the source image features and the downsampled reference features, a query operation is performed on the coordinates corresponding to the second subset using the photometric reference space to obtain a second set of target resolution pixel values, including: performing photometric alignment processing on the source image features and the downsampled reference features using the photometric reference space to generate photometric alignment features; and performing a query operation on the coordinates corresponding to the second subset based on the photometric alignment features to obtain a second set of target resolution pixel values.

[0049] As can be understood, photometric alignment refers to the process of matching and adjusting source image features with downsampled reference features at the representational level in photometric reference space to eliminate inconsistencies in illumination and color caused by exposure, white balance, or lens differences between two cameras (telephoto-wide-angle lens combination). Its purpose is to establish a consistent mapping of global and local brightness and color distribution between the two images (the image to be processed and the reference image). Photometric alignment features are the fused feature representations generated after photometric alignment. These features, while preserving the structure of the image to be processed, incorporate the photometric information of the reference image and can be used as input for subsequent coordinate queries and pixel reconstruction, ensuring that the reconstructed pixels have consistent illumination and color characteristics with the reference image.

[0050] Specifically, the photometric reference space receives source image features F from the feature extraction network. LR and downsampling reference feature F Ref 1× The two are then fused and modulated using a photometric alignment component to generate a unified photometric alignment feature F. Pho This step effectively alleviates the problem of inconsistent luminance distribution between images caused by differences in dual-camera hardware. Subsequently, this luminance alignment feature F... Pho Based on the query, for each coordinate point in the second subset, a query operation is performed through the multilayer perceptron corresponding to the photometric reference space (i.e., the second multilayer perceptron), mapping the corresponding RGB pixel values ​​point by point, ultimately forming the second set of target resolution pixel values. This set of pixels can maintain the content structure of the image to be processed while being highly coordinated with the reference image in terms of lighting and color representation, providing a crucial photometric consistency guarantee for achieving natural and realistic super-resolution effects.

[0051] As one possible implementation, in some embodiments, photometric alignment processing is performed on source image features and downsampled reference features using a photometric reference space to generate photometric aligned features. This includes: fusing source image features and downsampled reference features to obtain downsampled fused features; performing spatial pooling operations on texture aligned features and downsampled reference features respectively to obtain a first photometric representation of the image to be processed and a second photometric representation of the reference image; and modulating the downsampled fused features based on the first and second photometric representations to obtain photometric aligned features.

[0052] Understandably, the first photometric representation refers to the compact vector or low-dimensional feature map obtained by aggregating the features of the source image through spatial pooling. It is mainly used to characterize global illumination attributes such as average brightness and color distribution in the overall or local areas of the image, rather than local structural details. The second photometric representation refers to the statistical feature vector or low-dimensional feature map obtained by spatial pooling the downsampled reference features. It summarizes the global or local illumination attributes of the reference image at the same scale and is used to characterize the illumination state of the reference image.

[0053] Specifically, such as Figure 3 As shown in (a), since the three reference spaces perform their respective tasks in parallel, the photometric reference space receives source image features F from the feature extraction network. LR and downsampling reference feature F Ref 1× In addition, it can also receive texture alignment features F from the texture reference space. Tex First, the pre-defined super-resolution model can analyze the source image features F. LR and downsampling reference feature F Ref 1× Perform preliminary fusion processing to generate a downsampled fusion feature F. SR 1× Furthermore, the model can also separately address the texture alignment feature F. Tex and downsampling reference feature F Ref 1× Spatial pooling (e.g., global average pooling) is performed to extract their respective compact statistical information, generating a first photometric representation (from the texture alignment feature) and a second photometric representation (from the reference image). These two representations respectively carry the illumination statistics of the image to be processed and the reference image at their respective scales. The texture alignment feature F Tex It already contains texture-aligned detail information; using its photometric representation to guide the final fusion is more effective than directly using the source image features F. LR More accurate. Subsequently, based on the illumination information encoded by these two photometric representations (i.e., the illumination difference reflected by the first and second photometric representations), the downsampled and fused features F can be obtained through a designed photometric modulation component (e.g., a lightweight network containing convolutional layers and activation functions). SR 1× Modulation is performed, that is, the photometric modulation component learns the illumination mapping relationship between two images (the image to be processed and the reference image), and generates a set of modulation parameters accordingly. Based on these modulation parameters, the fused intermediate features (F... SR 1× Alternatively, the source image features can be adaptively adjusted directly to ultimately output the photometric alignment feature F. Pho .

[0054] It should be noted that, based on the source image features F LR and downsampling reference feature F Ref 1× The preset photometric alignment component of the super-resolution model can also input these two data points into an internal lightweight convolutional network, which learns the local spatial correspondence between them and outputs a downsampled index map P. 1× With downsampling confidence plot C 1× Among them, P 1× Predict a pointing downsampled reference feature F for each location Ref 1× The offset vector corresponding to the feature region, C 1× This characterizes the reliability of the offset matching. In the initial fusion stage, based on the downsampling index map P... 1× For the downsampled reference feature F Ref 1× Deformable sampling can extract features F from the source image. LR Spatially aligned reference feature blocks, then downsampled confidence map C 1× As weights, these feature blocks are compared with the source image feature F. LR Weighted fusion is performed. This process achieves adaptive selection and calibration among features, and the final output is an intermediate feature that retains the source image structure and incorporates precisely aligned reference information, namely the downsampled fusion feature F. SR 1× .

[0055] Optionally, in some embodiments, based on the upsampling features, a query operation is performed on the coordinates corresponding to the third subset using texture reference features to obtain a third set of target resolution pixel values, including: performing texture alignment processing on the photometric alignment features and upsampling features using texture reference space to generate texture alignment features; and performing a query operation on the coordinates corresponding to the third subset based on the texture alignment features to obtain a third set of target resolution pixel values.

[0056] As can be understood, texture alignment refers to the process of precisely mapping texture details in upsampled reference features to their corresponding positions in photometric alignment features using a non-rigid matching method in the texture reference space. This process aims to achieve high-fidelity local texture transfer and avoid texture distortion caused by misalignment or mismatch. Texture alignment features are the fused features generated after texture alignment. Building upon the color and lighting consistency provided by photometric alignment features, these features further integrate local texture details from the upsampled reference features, serving as a direct basis for detail enhancement in the final pixel reconstruction stage.

[0057] Specifically, the texture reference space first receives the photometric alignment feature F output from the preceding photometric reference space. Phoand the upsampled features F extracted from the reference image Ref 2× Next, the texture alignment component can perform texture alignment processing on these two sets of features, thereby generating a unified texture alignment feature F. Tex This feature not only inherits the color and lighting consistency from the photometric alignment feature but also incorporates high-resolution texture details from the reference image. Subsequently, using this texture alignment feature as the query basis, for each coordinate point in the third subset, a query operation is performed through a multilayer perceptron network dedicated to the texture reference space (i.e., the third multilayer perceptron), mapping out RGB pixel values ​​with clear texture details point by point, ultimately forming the third set of target resolution pixel values. This set of pixels significantly improves the local sharpness and visual realism of the reconstructed image and is a core component in achieving high-quality super-resolution results.

[0058] As one possible implementation, in some embodiments, texture alignment processing is performed on photometric alignment features and upsampling features using a texture reference space to generate texture alignment features, including: inputting the photometric alignment features and upsampling features into a preset dilated convolutional neural network to generate an upsampling index map and an upsampling confidence map; extracting target feature blocks from a reference image based on the upsampling index map; and performing weighted fusion of the target feature blocks based on the upsampling confidence map to obtain texture alignment features.

[0059] Understandably, the upsampled index map refers to a two-dimensional map predicted by a pre-defined dilated convolutional neural network based on input features. Each pixel's value corresponds to a coordinate offset or an index pointing to a spatial location in the upsampled reference features. This indicates which feature block in the upsampled reference features each position in the photometric alignment feature should be texture-aligned with. The upsampled confidence map refers to the map that is related to the index map. Figure 1 The generated two-dimensional map has a value for each pixel that represents the reliability or weight of the index matching result at that location. This value is used for weighting when fusing multiple feature blocks, with feature blocks at high confidence locations receiving higher weight in the fusion process. Target feature blocks refer to local feature regions extracted from the upsampled reference features by the index map. These feature blocks carry high-resolution texture details from the reference image and are a key source for texture reconstruction.

[0060] Specifically, such as Figure 3 As shown in (b), firstly, the photometric alignment feature F from the previous stage is... Pho Upsampled reference features F of the reference image Ref 2×The inputs are combined into a specially designed dilated convolutional neural network (DCNN). This network can utilize its large receptive field to capture long-range feature dependencies and predict two key outputs: an upsampled index map P. 2× Upsampling confidence mapping C 2× Among them, the upsampling index mapping graph P 2× In a learnable manner, it specifies which regions in the upsampled features each coordinate location should be texture-matched with; Upsampled confidence map C 2× This quantifies the reliability of each match. Next, based on the upsampled index mapping graph P... 2× The provided spatial guidance can be derived from the upsampled reference feature F Ref 2× A set of corresponding target feature blocks is extracted, and these feature blocks contain elements from the reference image I. Ref The rich texture details. Finally, combined with the upsampled confidence map C 2× Weighted fusion of each target feature block enhances high-confidence texture information while suppressing low-confidence information, thereby generating a texture alignment feature F that preserves the original structure and incorporates high-quality reference textures. Tex .

[0061] Therefore, by introducing a feature reselection mechanism in the texture reference space, this mechanism abandons the approach of relying on general feature extractors such as VGG. Instead, it generates pixel-level index maps and confidence maps through an independent convolutional network (i.e., a pre-defined dilated convolutional neural network). This enables more accurate selection and spatial alignment of texture features of the reference image, effectively eliminating interference caused by photometric inconsistencies, and thus significantly improving the texture accuracy and visual clarity of the reconstructed image.

[0062] like Figure 4 As shown, Figure 4This is a comparison of the visual effects of the image super-resolution reconstruction method of this application embodiment and representative methods in related technologies after super-resolution reconstruction at multiple test scales. The figure intuitively shows the output results of different methods when processing different scale magnification tasks in a real dual-camera scene for the same set of input low-resolution images (Low-Resolution, or LR) and reference images. Among them, MASA stands for Matching Acceleration and Spatial Adaptation, a matching acceleration and spatial adaptation mechanism; RCAN-CiaoS / R stands for Residual Channel Attention Networks, and CiaoS / R stands for Continuous Implicit Attention-in-Attention Network for Arbitrary-Scale Image Super-Resolution, a continuous implicit attention nested network for image super-resolution at arbitrary scales; TTSR stands for Texture Transformer for Super-Resolution, a texture transformer for super-resolution; DCSR stands for Dual-Camera Super-Resolution, a dual-camera super-resolution; SwinIR-DDIR stands for Swin Transformer for Image Restoration, an image restoration model based on the Swin Transformer, and DDIR stands for Dual-level Deformable Implicit Representation; GT stands for Ground Truth, the true value or high-resolution ground truth. The comparison shows that, while maintaining structural consistency and color naturalness, the method of this application significantly outperforms related technical solutions in terms of the clarity of reconstructed texture details, the sharpness of edges, and robustness against degradation of real-world images. The accompanying figure verifies, from a visual perception perspective, the effectiveness and superiority of the hybrid reference space structure proposed in this application's embodiments in handling arbitrary magnification and complex degradation dual-camera super-resolution tasks.

[0063] According to an embodiment of this application, a method for super-resolution reconstruction of images based on arbitrary magnification of dual-camera systems is proposed. This method determines the scale of the target resolution image based on the target magnification and generates a target resolution coordinate set corresponding to the scale. Based on the target resolution coordinate set, a preset super-resolution model is used to perform query and reconstruction operations on the image to be processed and a reference image to generate the target resolution image. The preset super-resolution model includes a null reference space, a photometric reference space, and a texture reference space. Therefore, by constructing and utilizing a hybrid reference space model containing null, photometric, and texture reference spaces, image features are queried and reconstructed in parallel across different levels of reference spaces. This solves the problems of poor generalization ability and photometric distortion or texture blurring in reconstructed images caused by fixed magnification and a single alignment space in related technologies. It achieves high-quality, high-fidelity super-resolution reconstruction of low-resolution images from real dual-camera systems at any user-specified magnification.

[0064] Next, referring to the accompanying drawings, an embodiment of this application provides a device for super-resolution reconstruction of images based on dual cameras at arbitrary magnification.

[0065] Figure 5 This is a block diagram of an embodiment of the present application of a dual-camera arbitrary magnification image super-resolution reconstruction device.

[0066] like Figure 5 As shown, the dual-camera arbitrary magnification image super-resolution reconstruction device 10 includes: an acquisition module 100, a first generation module 200, and a second generation module 300.

[0067] The acquisition module 100 is used to acquire the image to be processed and the reference image, wherein the resolution of the image to be processed is lower than the resolution of the reference image; The first generation module 200 is used to determine the scale of the target resolution image based on the target magnification and generate a set of target resolution coordinates corresponding to the scale. The second generation module 300 is used to perform query and reconstruction operations on the image to be processed and the reference image based on the target resolution coordinate set and using a preset super-resolution model to generate the target resolution image. The preset super-resolution model includes a null reference space, a photometric reference space and a texture reference space.

[0068] Optionally, in some embodiments, the second generation module 300 includes: The extraction unit is used to extract source image features and sampled reference features of the reference image of the image to be processed using the feature extraction network of the preset super-resolution model. The sampled reference features include downsampled reference features and upsampled reference features. The first processing unit is used to divide the target resolution coordinate set into first to third subsets according to a preset ratio, and based on the source image features, use the empty reference space to query the coordinates corresponding to the first subset to obtain the first set of target resolution pixel values. The second processing unit is used to perform a query operation on the coordinates corresponding to the second subset based on the source image features and downsampling reference features, and to obtain the second set of target resolution pixel values. The third processing unit is used to perform a query operation on the coordinates corresponding to the third subset based on the upsampling features and the texture reference features to obtain the third set of target resolution pixel values. The fusion unit is used to fuse the first set of target resolution pixel values, the second set of target resolution pixel values, and the third set of target resolution pixel values ​​to obtain a target resolution image.

[0069] Optionally, in some embodiments, the second processing unit includes: The first processing subunit is used to perform photometric alignment processing on the source image features and downsampled reference features using the photometric reference space to generate photometric aligned features. The second processing subunit is used to perform a query operation on the coordinates corresponding to the second subset based on the photometric alignment feature, so as to obtain the second set of target resolution pixel values.

[0070] Optionally, in some embodiments, the third processing unit is specifically used for: The third processing subunit is used to perform texture alignment processing on the photometric alignment features and upsampling features using the texture reference space to generate texture alignment features; The fourth processing subunit is used to perform a query operation on the coordinates corresponding to the third subset based on the texture alignment features to obtain the third set of target resolution pixel values.

[0071] Optionally, in some embodiments, the first processing subunit is specifically used for: By fusing source image features and downsampled reference features, the downsampled fused features are obtained; Spatial pooling operations are performed on the texture alignment features and the downsampled reference features respectively to obtain the first photometric representation of the image to be processed and the second photometric representation of the reference image. Based on the first and second photometric representations, the downsampling fusion features are modulated to obtain photometric alignment features.

[0072] Optionally, in some embodiments, the third processing subunit is specifically used for: The photometric alignment features and upsampling features are input into a pre-defined dilated convolutional neural network to generate an upsampling index map and an upsampling confidence map. Based on the upsampled index map, target feature blocks are extracted from the reference image; Based on the upsampled confidence map, the target feature blocks are weighted and fused to obtain texture alignment features.

[0073] It should be noted that the foregoing explanation of an embodiment of a dual-camera arbitrary magnification image super-resolution reconstruction method also applies to a dual-camera arbitrary magnification image super-resolution reconstruction device of this embodiment, and will not be repeated here.

[0074] According to an embodiment of this application, a dual-camera arbitrary magnification image super-resolution reconstruction device is proposed. It determines the scale of the target resolution image based on the target magnification and generates a target resolution coordinate set corresponding to the scale. Based on the target resolution coordinate set, a preset super-resolution model is used to perform query and reconstruction operations on the image to be processed and a reference image to generate the target resolution image. The preset super-resolution model includes a null reference space, a photometric reference space, and a texture reference space. Therefore, by constructing and utilizing a hybrid reference space model containing null, photometric, and texture reference spaces, image features are queried and reconstructed in parallel across different levels of reference spaces. This solves the problems of poor generalization ability and photometric distortion or texture blurring in reconstructed images caused by fixed magnification and a single alignment space in related technologies. It achieves high-quality, high-fidelity super-resolution reconstruction of low-resolution images from real dual-camera systems at any user-specified magnification.

[0075] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0076] When the processor 602 executes the program, it implements the super-resolution reconstruction method based on dual-camera arbitrary magnification images provided in the above embodiments.

[0077] Furthermore, electronic devices also include: Communication interface 603 is used for communication between memory 601 and processor 602.

[0078] The memory 601 is used to store computer programs that can run on the processor 602.

[0079] The memory 601 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0080] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0081] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0082] The processor 602 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0083] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for super-resolution reconstruction of images at arbitrary magnification using dual cameras.

[0084] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for super-resolution reconstruction of images at arbitrary magnification using dual cameras.

[0085] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0086] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0087] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for super-resolution reconstruction of images at arbitrary magnification using dual cameras, characterized in that, Includes the following steps: Obtain an image to be processed and a reference image, wherein the resolution of the image to be processed is lower than the resolution of the reference image; Based on the target magnification, the scale of the target resolution image is determined, and a set of target resolution coordinates corresponding to the scale is generated; Based on the target resolution coordinate set, a preset super-resolution model is used to perform query and reconstruction operations on the image to be processed and the reference image to generate the target resolution image. The preset super-resolution model includes a null reference space, a photometric reference space, and a texture reference space.

2. The method according to claim 1, characterized in that, The step of generating the target resolution image by performing query and reconstruction operations on the image to be processed and the reference image based on the target resolution coordinate set and using a preset super-resolution model includes: The feature extraction network of the preset super-resolution model is used to extract the source image features of the image to be processed and the sampled reference features of the reference image, wherein the sampled reference features include downsampled reference features and upsampled reference features; The target resolution coordinate set is divided into first to third subsets according to a preset ratio. Based on the source image features, the coordinates corresponding to the first subset are queried using the empty reference space to obtain the first set of target resolution pixel values. Based on the source image features and the downsampling reference features, the coordinates corresponding to the second subset are queried using the photometric reference space to obtain the second set of target resolution pixel values. Based on the upsampling features, the coordinates corresponding to the third subset are queried using the texture reference features to obtain the third set of target resolution pixel values. The target resolution image is obtained by fusing the first set of target resolution pixel values, the second set of target resolution pixel values, and the third set of target resolution pixel values.

3. The method according to claim 2, characterized in that, Based on the source image features and the downsampled reference features, the coordinates corresponding to the second subset are queried using the photometric reference space to obtain the second set of target resolution pixel values, including: The source image features and the downsampled reference features are photometrically aligned using the photometric reference space to generate photometric aligned features. Based on the photometric alignment feature, a query operation is performed on the coordinates corresponding to the second subset to obtain the second set of target resolution pixel values.

4. The method according to claim 3, characterized in that, The step of querying the coordinates corresponding to the third subset based on the upsampling features and using the texture reference features to obtain the third set of target resolution pixel values ​​includes: The photometric alignment feature and the upsampling feature are texture aligned using the texture reference space to generate a texture alignment feature. Based on the texture alignment features, a query operation is performed on the coordinates corresponding to the third subset to obtain the third set of target resolution pixel values.

5. The method according to claim 4, characterized in that, The step of performing photometric alignment processing on the source image features and the downsampled reference features using the photometric reference space to generate photometric aligned features includes: The source image features and the downsampled reference features are fused to obtain the downsampled fused features; Spatial pooling operations are performed on the texture alignment features and the downsampled reference features respectively to obtain the first photometric representation of the image to be processed and the second photometric representation of the reference image; Based on the first photometric characterization and the second photometric characterization, the downsampling fusion feature is modulated to obtain the photometric alignment feature.

6. The method according to claim 4, characterized in that, The step of performing texture alignment processing on the photometric alignment feature and the upsampling feature using the texture reference space to generate texture alignment features includes: The photometric alignment feature and the upsampling feature are input into a preset dilated convolutional neural network to generate an upsampling index map and an upsampling confidence map. Based on the upsampled index map, target feature blocks are extracted from the reference image; Based on the upsampled confidence map, the target feature block is weighted and fused to obtain the texture alignment feature.

7. A device for super-resolution reconstruction of images at arbitrary magnification based on dual cameras, characterized in that, include: An acquisition module is used to acquire an image to be processed and a reference image, wherein the resolution of the image to be processed is lower than the resolution of the reference image; The first generation module is used to determine the scale of the target resolution image based on the target magnification and generate a set of target resolution coordinates corresponding to the scale. The second generation module is used to perform query and reconstruction operations on the image to be processed and the reference image based on the target resolution coordinate set and using a preset super-resolution model to generate the target resolution image. The preset super-resolution model includes a null reference space, a photometric reference space and a texture reference space.

8. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement a dual-camera arbitrary magnification image super-resolution reconstruction method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement a dual-camera arbitrary magnification image super-resolution reconstruction method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The method includes a computer program, which, when executed by a processor, is used to implement the super-resolution reconstruction method for images based on dual cameras at arbitrary magnification as described in any one of claims 1-6.

Images