Image super-resolution method based on deep learning

Abstract

The invention provides an image super-division method based on deep learning, and belongs to the technical field of image super-division, and the method comprises the following steps: 1, preparing and preprocessing an image data set; 2, constructing a dynamic multi-modal feature fusion network model; step 3, constructing a progressive recognition framework; 4, training the network model and the progressive recognition framework through a self-supervised domain adaptive training strategy; 5, performing image reconstruction processing on the trained network model and the progressive recognition framework; and step 6, evaluating and testing the reconstructed image. According to the method, multi-modal dynamic fusion and physical perception progressive reconstruction are fused, cross-scale feature interaction, neural symbol reasoning and lightweight design are combined, high-quality and high-efficiency super-resolution reconstruction is achieved, detail generation is clearer, calculation efficiency is high, and the method conforms to a real scene of image practice.

Description

Technical Field

[0001] The present invention relates to the field of image super-resolution technology, and in particular to an image super-resolution method based on deep learning. Background Art

[0002] With the prevalence of high-definition displays, users are demanding higher and higher image quality. Limited by factors such as camera equipment, transmission bandwidth, and delivery strategies, the image quality displayed on terminals can be low in resolution, resulting in poor display quality. Image super-resolution is a technology that restores low-resolution input to high-resolution output, addressing these display quality issues.

[0003] Currently, the commonly used super-resolution method on the Internet is to encode and compress high-resolution images rendered in the cloud before sending them down. The terminal device side (i.e., client or mobile terminal) performs super-resolution and then displays high-resolution images. This method is called end-side super-resolution. Since the super-resolution algorithm has a large demand for computing resources, the super-resolution algorithm running on the terminal device has very high requirements for the hardware configuration of the terminal device. In related technologies, due to the limitations of terminal device hardware, problems may arise such as poor super-resolution effect, inability of super-resolution methods to be flexibly compatible with different configuration models, and high power consumption causing heat. Among them, related technologies generally target 540p super-resolution with a frame rate requirement of 30fps. However, in the current display of high-definition videos and cloud games, 720p super-resolution and 60fps are required, so further optimization of the super-resolution method is needed.

[0004] Traditional super-resolution methods are limited by the linear reconstruction assumption, and existing deep learning models (such as EDSR and Real-ESRGAN) suffer from the following issues: 1. Blurred detail generation: generating high-frequency details lacks physical plausibility; 2. Low computational efficiency: large-scale networks struggle to meet real-time requirements; and 3. Poor cross-scene generalization: relying on specific data distributions and lacking robustness to noise and artifacts. Therefore, it is necessary to design image super-resolution methods based on deep learning. Summary of the Invention

[0005] The purpose of the present invention is to provide an image super-resolution method based on deep learning to solve the technical problems mentioned in the background technology.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] An image super-resolution method based on deep learning, comprising the following steps:

[0008] Step 1: Image dataset preparation and preprocessing;

[0009] Step 2: Build a dynamic multimodal feature fusion network model;

[0010] Step 3: Build a progressive recognition framework;

[0011] Step 4: Train the network model and progressive recognition framework through a self-supervised domain adaptation training strategy;

[0012] Step 5: Use the trained network model and progressive recognition framework to perform image reconstruction;

[0013] Step 6: Evaluate and test the reconstructed image.

[0014] Furthermore, the specific process of step 1 is: constructing a dataset containing paired low-resolution and high-resolution images, normalizing the images, and performing data augmentation operations such as cropping and flipping as needed.

[0015] Furthermore, the specific process of step 2 is as follows: setting up a cross-modal knowledge distillation module, introducing auxiliary modalities as prior guidance, including depth maps, infrared images and textual semantic descriptions, extracting the edge structure of the depth map through the UNet network, and performing cross-attention fusion with the RGB features;

[0016] Through the synergy of physical imaging model and denoising prior, the super-resolution reconstruction process is constrained to conform to the real physical laws. A light field propagation model is set to simulate the imaging process of the optical system to constrain the physical rationality of the reconstruction results. Guided by the BM3D prior, the non-local similarity denoising algorithm is used to enhance the physical consistency of high-frequency details.

[0017] Furthermore, the specific process of step 3 is: progressive reconstruction, using a cascade structure to improve the resolution step by step. Each level contains local and global attention mechanisms, dynamic channel pruning and neural symbolic reasoning modules. The local and global attention mechanisms capture high-frequency details through local window attention, and the global cross attention fuses several scale contexts. Dynamic channel pruning realizes channel-level routing through Gumbel-Softmax, dynamically activates key paths according to input, and reduces redundant calculations. The neural symbolic reasoning module introduces a differential logic rule engine to perform symbolic reasoning on the structured features of edges and textures to suppress the generation of artifacts.

[0018] Furthermore, the specific process of step 4 is as follows: setting the adversarial domain obfuscation method, inserting domain randomization in the training phase, simulating different degradation conditions, including noise and compression artifacts, enhancing generalization ability, using the MAML algorithm to optimize the initial parameters so that the model can quickly adapt to unseen degradation types, and synthesizing training data based on the physical imaging model to cover complex degradations in real scenes.

[0019] Furthermore, the specific process of step 5 is as follows: first, annotate the image with text to obtain the text data of the image, then perform infrared processing on the image to obtain an infrared image, simulate the continuity of the natural image through the constrained gradient of the physical loss function, jointly optimize the fidelity and perceptual quality, and then reconstruct the image according to the model to obtain a reconstructed new image.

[0020] Furthermore, in step 6: measure the difference between the reconstructed image and the original image, the structural similarity between the two images,

[0021] The mean squared error between the θ and pixel values ​​was calculated, and the visual quality of the generated high-resolution images was evaluated by visual observation.

[0022] The present invention has the following beneficial effects due to the adoption of the above technical solution:

[0023] The present invention integrates multimodal dynamic fusion with physical perception progressive reconstruction, combines cross-scale feature interaction, neural symbolic reasoning and lightweight design, and achieves high-quality and efficient super-resolution reconstruction. The details are generated more clearly, the computational efficiency is high, and it conforms to the real scenes of image practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and by way of preferred embodiments. However, it should be noted that many of the details listed in this specification are merely provided to help the reader gain a thorough understanding of one or more aspects of the present invention, and these aspects of the present invention can be practiced even without these specific details.

[0026] like Figure 1 As shown, the image super-resolution method based on deep learning includes the following steps:

[0027] Step 1: Image dataset preparation and preprocessing. Construct a dataset containing paired low-resolution and high-resolution images. Normalize the images and perform data augmentation operations such as cropping and flipping as needed. Preprocessing also includes grayscale processing to achieve uniform image processing.

[0028] Step 2: Build a dynamic multimodal feature fusion network model. Set up a cross-modal knowledge distillation module and introduce auxiliary modalities as prior guidance. The auxiliary modalities include depth maps, infrared images, and textual semantic descriptions. The edge structure of the depth map is extracted through the UNet network and cross-attentionally fused with the RGB features.

[0029] Through the synergy of physical imaging model and denoising prior, the super-resolution reconstruction process is constrained to conform to the real physical laws. A light field propagation model is set to simulate the imaging process of the optical system to constrain the physical rationality of the reconstruction results. Guided by the BM3D prior, the non-local similarity denoising algorithm is used to enhance the physical consistency of high-frequency details.

[0030] Step 3: Build a progressive recognition framework. Progressive reconstruction uses a cascade structure to improve resolution step by step. Each level includes local and global attention mechanisms, dynamic channel pruning, and neural symbolic reasoning modules. The local and global attention mechanisms capture high-frequency details through local window attention, and global cross attention fuses multiple scale contexts. Dynamic channel pruning implements channel-level routing through Gumbel-Softmax, dynamically activates key paths based on input, reduces redundant calculations, and neural symbolic reasoning modules.

[0031] By introducing a differential logic rule engine, symbolic reasoning is performed on the structured features of edges and textures to suppress the generation of artifacts.

[0032] Step 4: Train the network model and progressive recognition framework using a self-supervised domain adaptation training strategy. Adversarial domain obfuscation is implemented, with domain randomization inserted during training to simulate different degradation conditions, including noise and compression artifacts, to enhance generalization. Initial parameters are optimized using the MAML algorithm, enabling the model to quickly adapt to unseen degradation types. Training data is synthesized based on a physical imaging model to cover complex degradations found in real-world scenarios.

[0033] Step 5: Use the trained network model and progressive recognition framework to reconstruct the image. First, annotate the image with text to obtain the image's text data. Then, perform infrared processing on the image to obtain an infrared image. Simulate the continuity of natural images through the constrained gradient of the physical loss function, jointly optimizing fidelity and perceptual quality. The image is then reconstructed based on the model to obtain a new reconstructed image.

[0034] Step 6: Evaluate and test the reconstructed image. We measure the difference between the reconstructed image and the original image, the structural similarity between the two images, and the mean squared error between pixel values. We also visually assess the visual quality of the generated high-resolution image.

[0035] Matters not covered by the present invention are known technologies.

[0036] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. Image super-resolution method based on deep learning, characterized by: The method comprises the following steps: Step 1: Image dataset preparation and preprocessing; Step 2: Build a dynamic multimodal feature fusion network model; Step 3: Build a progressive recognition framework; Step 4: Train the network model and progressive recognition framework through a self-supervised domain adaptation training strategy; Step 5: Use the trained network model and progressive recognition framework to perform image reconstruction; Step 6: Evaluate and test the reconstructed image.

2. The image super-resolution method based on deep learning according to claim 1, characterized in that: The specific process of step 1 is: construct a dataset containing paired low-resolution and high-resolution images, normalize the images, and perform data augmentation operations such as cropping and flipping as needed.

3. The image super-resolution method based on deep learning according to claim 1, characterized in that: The specific process of step 2 is as follows: setting up a cross-modal knowledge distillation module, introducing auxiliary modalities as prior guidance, including depth maps, infrared images, and textual semantic descriptions, extracting the edge structure of the depth map through the UNet network, and performing cross-attention fusion with the RGB features; Through the synergy of physical imaging model and denoising prior, the super-resolution reconstruction process is constrained to conform to the real physical laws. A light field propagation model is set to simulate the imaging process of the optical system to constrain the physical rationality of the reconstruction results. Guided by the BM3D prior, the non-local similarity denoising algorithm is used to enhance the physical consistency of high-frequency details.

4. The image super-resolution method based on deep learning according to claim 1, characterized in that: The specific process of step 3 is: progressive reconstruction, using a cascade structure to improve the resolution step by step. Each level contains local and global attention mechanisms, dynamic channel pruning and neural symbolic reasoning modules. The local and global attention mechanisms capture high-frequency details through local window attention, and the global cross attention fuses several scale contexts. Dynamic channel pruning implements channel-level routing through Gumbel-Softmax, dynamically activates key paths according to the input, and reduces redundant calculations. The neural symbolic reasoning module introduces a differential logic rule engine to perform symbolic reasoning on the structured features of edges and textures to suppress the generation of artifacts.

5. The image super-resolution method based on deep learning according to claim 1, characterized in that: The specific process of step 4 is as follows: setting the adversarial domain obfuscation method, inserting domain randomization in the training phase, simulating different degradation conditions, including noise and compression artifacts, enhancing generalization ability, using the MAML algorithm to optimize the initial parameters so that the model can quickly adapt to unseen degradation types, and synthesizing training data based on the physical imaging model to cover complex degradations in real scenes.

6. The image super-resolution method based on deep learning according to claim 1, characterized in that: The specific process of step 5 is as follows: first, annotate the image with text to obtain the text data of the image, then perform infrared processing on the image to obtain an infrared image, simulate the continuity of the natural image through the constrained gradient of the physical loss function, jointly optimize the fidelity and perceptual quality, and then reconstruct the image according to the model to obtain a reconstructed new image.

7. The image super-resolution method based on deep learning according to claim 1, characterized in that: In step 6: measure the difference between the reconstructed image and the original image, the structural similarity between the two images, and the mean squared error between the pixel values, and evaluate the visual quality of the generated high-resolution image by visually observing it.

Images