Lightweight image restoration method, device and medium based on multi-scale features

Abstract

This application discloses a lightweight image restoration method, device, and medium based on multi-scale features, relating to the field of image restoration. The method includes: inputting preprocessed image data as training samples into a neural network model to train the neural network model; extracting multi-scale features from the training samples using multiple convolution branches in the neural network model, and training the neural network model using a loss function; merging the multiple convolution branches in the neural network model through re-parameterization of convolution weights to obtain a single merged convolution branch; and performing image restoration using the merged neural network model. By introducing a multi-scale convolution structure, it is possible to effectively extract details and semantic information of the image under different receptive fields, enhancing the model's perception of complex textures and structures.

Description

Technical Field

[0001] The present application relates to the field of image restoration, and specifically to a lightweight image restoration method, device, and medium based on multi-scale features. Background Art

[0002] With the continuous advancement of the information society, images are playing an increasingly critical role in modern life and various professional applications. From daily recording on social platforms to image analysis in scientific research, image quality directly affects the transmission and understanding of information. However, due to limitations in imaging device resolution, transmission bandwidth bottlenecks, and storage resources, images often suffer from blurring, distortion, or loss of detail during acquisition, transmission, or storage. To improve image clarity and usability, image super-resolution technology has emerged as a key research direction for improving image quality.

[0003] Image super-resolution aims to restore low-definition images to high resolution, reconstructing their fine structure and details. This process uses algorithms to restore and predict image textures and edges, thereby improving image clarity. Traditional image upscaling methods, such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation, while simple to implement and fast to compute, often struggle to meet the demands of practical applications requiring high image quality due to their difficulty in restoring missing high-frequency information.

[0004] In recent years, with the advancement of deep learning, neural network-based super-resolution methods have become a research hotspot. Compared to traditional methods, these techniques can learn complex image features and contextual relationships, producing clear images while preserving more detail. They demonstrate broad applicability and potential in a variety of fields. For example, in medical image processing, high-definition images help doctors make more accurate diagnoses and analyses; in intelligent security systems, improved surveillance footage facilitates the identification of key targets and behaviors; and super-resolution also holds significant technical value in cultural heritage preservation, film and television restoration, and virtual reality.

[0005] However, despite the breakthroughs made by deep learning in image super-resolution, problems such as image artifacts, insufficient detail recovery, and poor visual consistency still exist. When faced with complex scenes and high-resolution target images, effectively reconstructing realistic and detailed image structures while ensuring ease of deployment in real-world environments remains a major challenge facing current research. Summary of the Invention

[0006] To solve the above problems, this application proposes a lightweight image restoration method based on multi-scale features, including:

[0007] Obtaining a data set for model training and preprocessing image data in the data set;

[0008] Inputting the preprocessed image data as training samples into the neural network model to train the neural network model;

[0009] Extracting multi-scale features from the training samples using the neural network model using multiple convolution branches, and training the neural network model using a loss function; wherein the convolution kernels in different convolution branches have different sizes;

[0010] Test the trained neural network model through test samples;

[0011] Merging multiple convolution branches in the neural network model by re-parameterizing the convolution weights to obtain a single merged convolution branch;

[0012] Image restoration is performed using the combined neural network model.

[0013] In one example, preprocessing the image data in the dataset specifically includes:

[0014] Performing image segmentation on the image data in the data set to obtain a plurality of overlapping sub-regions;

[0015] Performing data enhancement processing on the overlapping sub-region to obtain an enhanced sub-region;

[0016] The enhancer region is randomly cropped to obtain the corresponding image block.

[0017] In one example, the neural network model is used to extract multi-scale features from the training sample using multiple convolution branches, and the neural network model is trained using a loss function, specifically including:

[0018] Inputting the training samples into the neural network model; wherein the training samples include original image data and degraded image data;

[0019] Extract features of the degraded image data through multiple parallel convolution branches in the shallow convolution layer of the neural network model, expand the feature dimension of the feature map output by each convolution branch, and output a corresponding high-dimensional feature map;

[0020] The high-dimensional feature maps are fused through a hybrid attention mechanism to obtain multi-scale enhanced features; wherein the hybrid attention mechanism includes at least channel attention and global attention;

[0021] Compressing the multi-scale enhancement features and restoring the resolution by upsampling to obtain reconstructed image data;

[0022] Based on the loss function, the reconstructed image data and the original image data are compared, and the model parameters of the neural network model are optimized through back propagation.

[0023] In one example, the high-dimensional feature maps are fused through a hybrid attention mechanism to obtain multi-scale enhanced features, specifically including:

[0024] For the high-dimensional feature map output by each convolution branch, a lightweight convolution module is used to extract the regional feature mask map corresponding to the high-dimensional feature map; the regional feature mask map includes a high-frequency area and a low-frequency area;

[0025] For the regional feature mask map, setting the attention weight corresponding to each attention mechanism in the hybrid attention mechanism;

[0026] According to the regional characteristic mask map corresponding to each high-dimensional feature map, the attention collaborative weight map is generated through multi-branch mask collaboration;

[0027] Based on the attention collaborative weight map, the high-dimensional feature map is fused through a hybrid attention mechanism to obtain multi-scale enhanced features.

[0028] In one example, based on the regional feature mask maps corresponding to each high-dimensional feature map, an attention coordination weight map is generated through multi-branch mask collaboration, specifically including:

[0029] The regional feature mask maps corresponding to each high-dimensional feature map are stacked into a mask tensor;

[0030] For the mask tensor, determine the frequency weight maps corresponding to the high-frequency area and the low-frequency area respectively by averaging the mask, and generate an attention coordination weight map based on the frequency weight maps;

[0031] Among them, the higher the weight of the high-frequency area in the frequency weight map, the higher the weight of the global attention in the attention coordination weight map; the higher the weight of the low-frequency area in the frequency weight map, the higher the weight of the channel attention in the attention coordination weight map.

[0032] In one example, the method further includes:

[0033] For the degraded image data, a corresponding edge region is detected by an edge detection operator, and the edge region is repaired by a lightweight convolution module;

[0034] For the edge area, a similarity comparison is performed with the high-frequency area in each high-dimensional feature map, and the edge weight corresponding to each high-dimensional feature map is determined according to the similarity comparison result;

[0035] Based on the edge weight, the mask tensor is modified to delete, in the mask tensor, high-frequency regions corresponding to the high-dimensional feature maps whose edge weights are lower than a preset threshold;

[0036] A weighted sum is performed on the high-frequency area corresponding to the frequency weight map through the edge area to modify the frequency weight map.

[0037] In one example, multiple convolution branches in the neural network model are merged by re-parameterizing the convolution weights to obtain a single merged convolution branch, specifically including:

[0038] For each convolution branch in the neural network model, determining the corresponding convolution kernel size and bias term;

[0039] Aligning the convolution kernel sizes and summing them to obtain an equivalent convolution kernel; and summing the bias terms to obtain an equivalent bias term;

[0040] Based on the equivalent convolution kernel and the equivalent bias term, a single merged convolution branch is obtained.

[0041] In one example, the convolution kernel sizes are aligned and summed to obtain an equivalent convolution kernel, specifically including:

[0042] Aligning the convolution kernel sizes to obtain aligned convolution kernels;

[0043] According to the convolution kernel sizes before alignment, the convolution branches are grouped so that the maximum difference in the convolution kernel sizes in each group is lower than a preset difference;

[0044] For each group, the aligned convolution kernels are summed to obtain the sub-equivalent convolution kernel;

[0045] For each group, perform external summation on each sub-equivalent convolution kernel to obtain the equivalent convolution kernel.

[0046] On the other hand, this application also proposes a lightweight image restoration device based on multi-scale features, including:

[0047] at least one processor; and,

[0048] a memory communicatively connected to the at least one processor; wherein,

[0049] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the lightweight image restoration method based on multi-scale features as described in any of the above examples.

[0050] On the other hand, the present application also proposes a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to implement a lightweight image restoration method based on multi-scale features as described in any of the above examples.

[0051] The lightweight image restoration method based on multi-scale features proposed in this application can bring the following beneficial effects:

[0052] 1. By introducing a multi-scale convolutional structure, it can effectively extract image details and semantic information in different receptive fields, enhancing the model's perception of complex textures and structures. At the same time, combined with an efficient channel attention mechanism, it achieves feature selection and information enhancement, controlling the number of parameters and computational overhead while improving image reconstruction quality.

[0053] 2. The convolution weight re-parameterization strategy is adopted to make the model more expressive during the training phase. During inference deployment, it can be converted into a lightweight single-branch equivalent structure, which improves inference speed and resource utilization, and is suitable for efficient computing needs in actual deployment environments.

[0054] 3. Through the collaborative processing of multi-stage feature fusion and attention mechanism, as well as efficient upsampling design, the texture information and structural coherence of the image are effectively preserved. Even when faced with extremely low-resolution or degraded images, clear, natural and realistic high-resolution images can still be reconstructed, which is superior to traditional super-resolution methods and some existing deep learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0056] Figure 1 Schematic diagram of the process of a lightweight image restoration method based on multi-scale features in an embodiment of the present application;

[0057] Figure 2 This is a schematic diagram of a lightweight image restoration method based on multi-scale features in one scenario in an embodiment of the present application;

[0058] Figure 3 This is a schematic diagram of the structure of multiple convolution branches in one scenario in an embodiment of the present application;

[0059] Figure 4 Schematic diagram of an equivalent convolution kernel in one case in an embodiment of the present application;

[0060] Figure 5 Schematic diagram of a lightweight image restoration device based on multi-scale features in an embodiment of the present application. DETAILED DESCRIPTION

[0061] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0062] The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0063] Traditional image super-resolution technologies, whether relying on traditional interpolation methods or deep learning models, often struggle to maintain a balance between overall semantic consistency, structural integrity, and detailed texture restoration for extremely low-resolution or severely degraded images. Especially when performing large-scale magnification operations, these methods often struggle to balance image fidelity and clarity, prone to blurring, loss of detail, or structural distortion, severely impacting the image's practicality in high-precision scenarios.

[0064] Based on this, Figure 1 As shown, the embodiment of the present application provides a lightweight image restoration method based on multi-scale features, including:

[0065] S101: Obtain a data set for model training, and preprocess the image data in the data set.

[0066] The dataset can be constructed by local collection, or by collecting images through the network, or by using some already constructed databases, such as Figure 2 As shown in the figure, the DF2K dataset is used as the dataset for model training. The DF2K dataset is a comprehensive dataset designed for image super-resolution tasks. It is composed of two high-resolution image datasets, DIV2K and Flickr2K. Its core goal is to provide rich and diverse samples for training and testing image super-resolution algorithms.

[0067] Before the image is input into the neural network model, the DF2K dataset needs to be adapted because the original image has a high 2K resolution and the computational overhead is too high to use it directly for training.

[0068] Specifically, the image data in the data set is divided into multiple overlapping sub-regions, for example, the image data in the data set is divided into multiple overlapping sub-images with a size of 480×480.

[0069] Data augmentation is performed on the overlapping subregions to obtain enhanced subregions. Data augmentation can include horizontal or vertical mirroring operations and combined with random rotation angles (such as 90°, 180°, 270°, etc.) to achieve data augmentation.

[0070] Randomly crop the enhancer region to obtain the corresponding image patch. For example, randomly select a 48×48 patch from the enhancer region and use it as the input sample for model training.

[0071] During actual training, 800 high-resolution images from the public DIV2K dataset were selected as the training sample base. Data augmentation techniques such as image cropping, horizontal flipping, and rotation were used to expand the training set to improve the model's generalization capabilities. During network training, a batch size of 32 was set, and the total number of training iterations reached 500,000. Model construction and training were completed using the PyTorch deep learning framework.

[0072] S102: The pre-processed image data is used as a training sample and input into a neural network model to train the neural network model.

[0073] The neural network model can choose a model architecture such as RepNet model, convolutional neural network model, etc. The preprocessed image dataset is input into the neural network model, for example, Figure 2 As shown in FIG, when the RepNet model is selected, the RepNet network is used for iterative training to obtain the image data and pre-training model after training.

[0074] S103: Using the neural network model, using multiple convolution branches, extracting multi-scale features from the training samples, and training the neural network model through a loss function; wherein the convolution kernel sizes in different convolution branches are different.

[0075] During model training, training samples are fed into the neural network model. These training samples include original image data and degraded image data. Original image data refers to pre-processed image data, while degraded image data is obtained by blurring, compressing, and other processing.

[0076] By using multiple convolution branches in parallel in the shallow convolutional layers of the neural network model, features of the degraded image data are extracted, and the feature dimension of the feature map output by each convolution branch is expanded to output the corresponding high-dimensional feature map. Among them, multiple convolution branches can constitute multi-scale convolution feature extraction, which is an enhancement strategy in image super-resolution reconstruction and is used to more comprehensively capture local and global information in the image. By using convolution kernels of different sizes in parallel (for example, 3×3, 5×5, 7×7, etc.), feature information under different receptive fields in the image is extracted, thereby achieving a multi-level characterization of image details and structures. Smaller convolution kernels are more suitable for capturing detailed features such as edges and textures, while larger convolution kernels help to extract wider-area contextual information.

[0077] Multi-scale feature extraction is designed to improve feature perception in image super-resolution. As an important component of the overall network, its goal is to capture the texture and structural information of images at different spatial scales without significantly increasing parameter overhead, thereby enhancing the model's detail recovery and context understanding capabilities.

[0078] Convolution kernels of various sizes are used in parallel to process the input feature map, extracting multi-scale semantic information through different receptive fields. Small convolution kernels (for example, 3×3) help capture fine texture and edge details, while large convolution kernels (for example, 5×5 or 7×7) are better at extracting large-scale structural features. This multi-scale parallel approach enables the model to perceive image features at multiple levels, from local to global, forming a richer and more robust feature representation.

[0079] like Figure 3 As shown, given the input feature X, the calculation process of the multi-scale feature extraction module is shown in formula (1) to formula (6):

[0080] Formula (1);

[0081] Formula (2);

[0082] Formula (3);

[0083] Formula (4);

[0084] Formula (5);

[0085] Formula (6).

[0086] in, It refers to executing the corresponding operator on the input feature X, and a is the convolution kernel size. When a=9, the convolution kernel size is 9×9. Similarly, a can be equal to 3, 5, or 9. dn refers to the corresponding expansion rate. When n=1, the expansion rate is 1. Similarly, n can be equal to 1, 2, 3, or 4.

[0087] Convolutional features are extracted in the shallow convolutional layer of the model to refine basic image information and expand the feature dimension to obtain a high-dimensional feature map. Each convolution branch can extract the corresponding high-dimensional feature map, thus laying the foundation for deep analysis.

[0088] At this point, a hybrid attention mechanism is used to fuse high-dimensional feature maps to generate multi-scale enhanced features. This hybrid attention mechanism includes at least channel attention and global attention. During the fusion phase, features extracted at each scale are combined into a unified feature representation through concatenation or weighted fusion, facilitating further network processing. This multi-scale feature fusion mechanism is particularly effective in image reconstruction tasks, improving the model's ability to perceive complex textures and structural changes.

[0089] The Hybrid Attention Mechanism is a method that enhances feature expressiveness by simultaneously modeling inter-channel relationships and spatial position relationships. It uses two complementary attention mechanisms, channel attention and global attention, to optimize the semantic and spatial information of multi-scale features.

[0090] For each high-dimensional feature map, corresponding attention processing is performed through channel attention and global attention respectively to obtain the corresponding channel-weighted features and spatial-weighted features. The channel-weighted features and spatial-weighted features are then fused to obtain the fused features corresponding to the high-dimensional feature map. The fused features corresponding to all high-dimensional feature maps are then fused (for example, by splicing or weighted fusion) to obtain multi-scale enhanced features.

[0091] The multi-scale enhanced features are compressed and the resolution is restored through upsampling to obtain reconstructed image data. Based on the loss function, the reconstructed image data is compared with the original image data, and the model parameters of the neural network model are optimized through backpropagation.

[0092] The loss function includes the L1 loss function, which is used to guide the network to restore image details more accurately. The L1 loss function definition can be shown as formula (7):

[0093] Formula (7);

[0094] in represents the super-resolution reconstructed image data, represents high-resolution raw image data, while Represents the L1 norm. This loss function emphasizes pixel-level accuracy and helps the model reconstruct clearer image edges in super-resolution tasks.

[0095] Furthermore, although the above scheme can enhance features through the hybrid attention mechanism, it does not optimize the differences between high-frequency details (such as textures and edges) and low-frequency structures (such as smooth areas) in image restoration tasks, which may result in a waste of computing resources and insufficient accuracy in selecting key areas in complex degradation scenarios.

[0096] Based on this, for the high-dimensional feature map output by each convolution branch, the regional feature mask map corresponding to the high-dimensional feature map is extracted through a lightweight convolution module; the regional feature mask map includes high-frequency areas and low-frequency areas.

[0097] The lightweight convolution module can be configured as a three-layer network with small convolution kernels to quickly parse the input feature map. In the region feature mask, each pixel is marked as belonging to a high-frequency region (corresponding to complex textures and large gradient changes, such as edges and textures) or a low-frequency region (corresponding to smooth colors and small gradient changes, such as the sky and solid background).

[0098] For the regional feature mask map, set the attention weights corresponding to each attention mechanism in the hybrid attention mechanism.

[0099] In high-frequency regions, spatial attention dominates, giving global attention a higher weight. Position-sensitive convolution or spatial pooling are used to calculate the weights of each spatial position in the high-dimensional feature map, thereby enhancing the local detail response in high-frequency regions. For example, the weights of edge pixels are increased to ensure accurate reconstruction of texture details and avoid blurring.

[0100] In high-frequency areas, whether channel attention is needed for corresponding processing can be set based on demand. Even if processing is required, its processing weight is lower than that of global attention. For example, the high-frequency areas are processed separately by global attention and channel attention, and then weighted fusion is performed according to the attention weights corresponding to each attention mechanism (among which the attention weight of global attention is higher).

[0101] For low-frequency regions, channel attention dominates, meaning it carries a higher weight. Global pooling and fully connected layers are used to calculate the weights of each channel, adjusting the feature ratio between channels, optimizing overall color consistency, maintaining visual coherence in large, smooth areas, and suppressing noise. Similarly, in low-frequency regions, global attention processing can be configured based on demand. Even if processing is required, its weight is lower than that of channel attention.

[0102] Based on the regional feature mask maps corresponding to each high-dimensional feature map, an attention coordination weight map is generated through multi-branch mask collaboration. In the regional feature mask map, the closer the value of each point is to 1, the higher the frequency, the higher the weight of global attention in the attention coordination weight map, and the lower the weight of channel attention. Conversely, the closer the value of a point is to 0, the lower the frequency, the lower the weight of global attention, and the higher the weight of channel attention.

[0103] The regional characteristic mask maps corresponding to each high-dimensional feature map are stacked into a mask tensor. For the mask tensor, the frequency weight maps corresponding to the high-frequency and low-frequency regions are determined by averaging the masks, and an attention coordination weight map is generated based on the frequency weight maps. For example, for each point, the mask tensor contains multiple stacked regional characteristic mask maps. Based on the value of the point in each regional characteristic mask map, the average is calculated to obtain the value of the point in the mask tensor, thereby obtaining the frequency weight maps corresponding to the high-frequency and low-frequency regions, respectively. Then, in a similar manner, the attention coordination weight map is generated based on the value of each point in the frequency weight map.

[0104] Each region can be composed of a fixed number of points. The average of the values ​​of each point in the region is used to obtain a value representing the overall situation of the region. When the value is greater than 0.5, the region is considered to be a high-frequency region, and the higher the value, the higher the weight of the high-frequency region. When the value is less than 0.5, the region is considered to be a low-frequency region, and the lower the value, the higher the weight of the low-frequency region. When the value is equal to 0.5, it can also be set as a high-frequency region, a low-frequency region, or a separate type of region based on needs.

[0105] Moreover, the higher the weight of the high-frequency area in the frequency weight map, the higher the weight of the global attention in the attention coordination weight map; the higher the weight of the low-frequency area in the frequency weight map, the higher the weight of the channel attention in the attention coordination weight map.

[0106] At this time, the obtained attention coordination weight map represents the weights of each attention mechanism corresponding to each high-frequency area and low-frequency area. Based on the attention coordination weight map, the high-dimensional feature map is fused through the hybrid attention mechanism to obtain multi-scale enhanced features.

[0107] Furthermore, edge reinforcement injection can be performed to force the model to preserve the original structure contours during the restoration process and prevent the attention mechanism from over-smoothing key edges.

[0108] During edge enhancement injection, the edge detection operator is used to detect the corresponding edge region in the degraded image data, and then the edge region is repaired using a lightweight convolution module. The edge detection operator can be a Sobel operator or a Canny operator. The image gradient is calculated using the edge detection operator, and a contour response map is extracted. Points in the contour response map with a gradient greater than a threshold are marked as edges, thereby obtaining the corresponding edge region.

[0109] However, the edges extracted by traditional operators are broken or noisy. Therefore, a lightweight convolution module (for example, a three-layer lightweight convolution module, including 1×1, 3×3, and 1×1 convolution layers) is used to connect the broken edges, filter out noise points, and generate a smooth edge area.

[0110] For the edge area, a similarity comparison is performed with the high-frequency area in each high-dimensional feature map, and the edge weight corresponding to each high-dimensional feature map is determined based on the similarity comparison result.

[0111] Because different branches may conflict in their responses to the same edge, for example, a small core branch detects fine edges, while a large core branch tends to ignore them. Therefore, each convolution branch corresponds to a high-dimensional feature map, and the similarity between the edge region and the high-frequency regions in each high-dimensional feature map is calculated. When there are multiple edge regions, the similarity corresponding to each edge region is calculated separately. Of course, when calculating the similarity between an edge region and a high-frequency region, it is only necessary to calculate the similarity between the edge region and the high-frequency region where it is located.

[0112] Based on the edge weights, the mask tensor is modified to remove high-frequency regions corresponding to high-dimensional feature maps with edge weights below a preset threshold. For high-frequency regions with low edge weights, the convolution branch corresponding to the high-frequency region is considered to have a low degree of edge detection. Therefore, the high-frequency region corresponding to the edge in the high-dimensional feature map corresponding to the convolution branch is removed from the mask tensor.

[0113] After determining the corresponding frequency weight map (which contains the distribution positions of each high-frequency area and low-frequency area) through the mask tensor, the frequency weight map can be corrected by performing weighted summation on the edge area and the high-frequency area corresponding to the frequency weight map, thereby forcing the model to focus on the contour.

[0114] Of course, if there are multiple edge areas, for each edge area, a weighted sum is performed based on the value of each point at its location (in the edge area, its value is 1) and the value of the high-frequency area corresponding to the point in the frequency weight map (the highest value is 1). Usually, the value of the point in the frequency weight map can be increased to enhance the contour.

[0115] S104: Testing the trained neural network model through test samples.

[0116] When evaluating the performance of trained neural network models (such as RepNet models), standard test datasets such as Set5, Set14, BSD100, Urban100, and Manga109 can be used. This testing involves both qualitative and quantitative analysis. Qualitative analysis primarily assesses the model's performance in detail restoration and image quality through image visual comparison. Quantitative analysis uses metrics such as PSNR and SSIM to measure the model's accuracy and stability in image super-resolution tasks.

[0117] S105: Merging multiple convolution branches in the neural network model through convolution weight re-parameterization to obtain a single merged convolution branch.

[0118] Convolution weight reparameterization is a technical approach to separate parameter expression during network training and inference phases, aiming to optimize the model's structural efficiency and inference performance while maintaining its expressiveness. This method uses a more flexible structure during the training phase to improve the model's fitting ability, such as combining multiple different convolution kernels, batch normalization layers (BatchNormalization) or nonlinear activation functions. Figure 3 As shown in the figure, by designing five parallel convolution branches with different convolution kernel sizes and dilation rates, it is possible to effectively cover multi-scale information from local details to global structures.

[0119] After training is completed, in the inference phase, mathematical methods are used to fuse these complex structures, remapping (or merging) them into an equivalent, simpler convolution operation (for example, a single equivalent convolution kernel), retaining only the pure convolution layer for inference, thereby reducing the amount of computation and memory usage and improving the execution efficiency of the model at the deployment end.

[0120] During inference, reparameterization techniques are used to fuse these five branches into an equivalent single-branch convolutional structure, transforming multiple convolutional paths into a single one, reducing the number of parameters and computational overhead. This optimizes deployment efficiency while maintaining model accuracy, making it suitable for real-time and resource-sensitive applications.

[0121] Specifically, for each convolution branch in the neural network model, determine the corresponding convolution kernel size and bias term. Align the convolution kernel sizes and sum them to obtain the equivalent convolution kernel. For example, Figure 4 As shown, the five convolution kernels are aligned to a 9×9 convolution kernel size. For example, zeros are added to the periphery of the convolution kernel to achieve alignment.

[0122] The bias terms are summed to obtain the equivalent bias terms. Based on the equivalent convolution kernel and the equivalent bias terms, the two are added together to obtain a single merged convolution branch. This process can be shown as formula (8):

[0123] Formula (8);

[0124] in, is a single convolution branch after the convolution weight is re-parameterized, Represents the convolution branches before the convolution weights are re-parameterized, are equivalent convolution kernels aligned to the same size, is the sum of the equivalent convolution kernels of all convolution branches, Represents the convolution operation, and finally all Merge into a large equivalent convolution kernel, represents the bias term, Then it is the equivalent bias term obtained by summing all bias terms.

[0125] Furthermore, in order to avoid the failure of equivalent convolution kernels after merging multiple branches, the convolution kernels can be grouped and internally summed, and then externally summed between groups.

[0126] Align the convolution kernel sizes to obtain aligned convolution kernels. This process is similar to the alignment process above.

[0127] According to the convolution kernel sizes before alignment, the convolution branches are grouped so that the maximum difference in the convolution kernel sizes in each group is less than the preset difference. For example, 3×3 and 5×5 are grouped together, 7×7 and 9×9 are grouped together, or the convolution kernel settings are as follows: Figure 3 As shown, group all 3×3 into one group, group all 5×5 into one group, and group all 9×9 into one group.

[0128] Within each group, the aligned convolution kernels are summed to obtain the sub-equivalent convolution kernel; outside each group, the sub-equivalent convolution kernels are summed to obtain the equivalent convolution kernel, thereby avoiding weight distribution imbalance and enhancing reparameterization stability.

[0129] S106: Perform image restoration using the merged neural network model.

[0130] Taking the RepNet model as an example, the RepNet model is a deep learning network designed for image super-resolution tasks. It combines the efficient modeling capabilities of convolutional neural networks with the global perception properties of attention mechanisms. Starting with a shallow feature extraction module, standard convolutional layers are used to acquire basic visual information from the input low-resolution image. Within the deep structure, the RepNet model introduces a hybrid mechanism that fuses channel attention and self-attention, effectively enhancing responsiveness to key semantic regions while balancing global context and local texture representation.

[0131] To further improve model performance, RepNet leverages reparameterization technology to model multiple large-kernel convolution branches with different receptive fields in parallel during training. These branches are then fused into a single convolution operation during inference, achieving both expressiveness and computational efficiency. This introduces richer contextual information perception without increasing inference costs. The feature-processed tensor is then restored to a high-resolution image using an upsampling module, where detailed reconstruction is achieved through pixel shuffling or deconvolution.

[0132] During training, the RepNet model uses the L1 loss function as its optimization objective to enhance the quality of edge information restoration. Overall, the RepNet model in this application achieves efficient and accurate super-resolution reconstruction through the synergy of a re-parameterized large-kernel convolution strategy and an attention mechanism.

[0133] 1. By introducing a multi-scale convolutional structure, it can effectively extract image details and semantic information in different receptive fields, enhancing the model's perception of complex textures and structures. At the same time, combined with an efficient channel attention mechanism, it achieves feature selection and information enhancement, controlling the number of parameters and computational overhead while improving image reconstruction quality.

[0134] 2. The convolution weight re-parameterization strategy is adopted to make the model more expressive during the training phase. During inference deployment, it can be converted into a lightweight single-branch equivalent structure, which improves inference speed and resource utilization, and is suitable for efficient computing needs in actual deployment environments.

[0135] 3. Through the collaborative processing of multi-stage feature fusion and attention mechanism, as well as efficient upsampling design, the texture information and structural coherence of the image are effectively preserved. Even when faced with extremely low-resolution or degraded images, clear, natural and realistic high-resolution images can still be reconstructed, which is superior to traditional super-resolution methods and some existing deep learning models.

[0136] like Figure 5 As shown, the embodiment of the present application also proposes a lightweight image restoration device based on multi-scale features, including:

[0137] at least one processor; and,

[0138] a memory communicatively connected to the at least one processor; wherein,

[0139] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the lightweight image restoration method based on multi-scale features as described in any of the above embodiments.

[0140] An embodiment of the present application further proposes a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to implement the lightweight image restoration method based on multi-scale features as described in any of the above embodiments.

[0141] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0142] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0143] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A lightweight image restoration method based on multi-scale features, characterized in that: include: Obtaining a data set for model training and preprocessing image data in the data set; Inputting the preprocessed image data as training samples into the neural network model to train the neural network model; Extracting multi-scale features from the training samples using the neural network model using multiple convolution branches, and training the neural network model using a loss function; wherein the convolution kernels in different convolution branches have different sizes; Test the trained neural network model through test samples; Merging multiple convolution branches in the neural network model by re-parameterizing the convolution weights to obtain a single merged convolution branch; Image restoration is performed using the combined neural network model; The neural network model is used to extract multi-scale features from the training samples using multiple convolution branches, and the neural network model is trained using a loss function, specifically including: Inputting the training samples into the neural network model; wherein the training samples include original image data and degraded image data; Extract features of the degraded image data through multiple parallel convolution branches in the shallow convolution layer of the neural network model, expand the feature dimension of the feature map output by each convolution branch, and output a corresponding high-dimensional feature map; The high-dimensional feature maps are fused through a hybrid attention mechanism to obtain multi-scale enhanced features; wherein the hybrid attention mechanism includes at least channel attention and global attention; Compressing the multi-scale enhancement features and restoring the resolution by upsampling to obtain reconstructed image data; Based on a loss function, the reconstructed image data and the original image data are compared, and model parameters of the neural network model are optimized through back propagation; The high-dimensional feature maps are fused through a hybrid attention mechanism to obtain multi-scale enhanced features, specifically including: For the high-dimensional feature map output by each convolution branch, a lightweight convolution module is used to extract the regional feature mask map corresponding to the high-dimensional feature map; the regional feature mask map includes a high-frequency area and a low-frequency area; For the regional feature mask map, setting the attention weight corresponding to each attention mechanism in the hybrid attention mechanism; According to the regional characteristic mask map corresponding to each high-dimensional feature map, the attention collaborative weight map is generated through multi-branch mask collaboration; Based on the attention collaborative weight map, the high-dimensional feature map is fused through a hybrid attention mechanism to obtain multi-scale enhanced features; According to the regional feature mask map corresponding to each high-dimensional feature map, the attention collaborative weight map is generated through multi-branch mask collaboration, specifically including: The regional feature mask maps corresponding to each high-dimensional feature map are stacked into a mask tensor; For the mask tensor, determine the frequency weight maps corresponding to the high-frequency area and the low-frequency area respectively by averaging the mask, and generate an attention coordination weight map based on the frequency weight maps; Among them, the higher the weight of the high-frequency area in the frequency weight map, the higher the weight of the global attention in the attention coordination weight map; the higher the weight of the low-frequency area in the frequency weight map, the higher the weight of the channel attention in the attention coordination weight map.

2. The lightweight image restoration method based on multi-scale features according to claim 1 is characterized in that: Preprocessing the image data in the dataset specifically includes: Performing image segmentation on the image data in the data set to obtain a plurality of overlapping sub-regions; Performing data enhancement processing on the overlapping sub-region to obtain an enhanced sub-region; The enhancer region is randomly cropped to obtain the corresponding image block.

3. The lightweight image restoration method based on multi-scale features according to claim 1, characterized in that: The method further comprises: For the degraded image data, a corresponding edge region is detected by an edge detection operator, and the edge region is repaired by a lightweight convolution module; For the edge area, a similarity comparison is performed with the high-frequency area in each high-dimensional feature map, and the edge weight corresponding to each high-dimensional feature map is determined according to the similarity comparison result; Based on the edge weight, the mask tensor is modified to delete, in the mask tensor, high-frequency regions corresponding to the high-dimensional feature maps whose edge weights are lower than a preset threshold; A weighted sum is performed on the high-frequency area corresponding to the frequency weight map through the edge area to modify the frequency weight map.

4. The lightweight image restoration method based on multi-scale features according to claim 1, characterized in that: By re-parameterizing the convolution weights, multiple convolution branches in the neural network model are merged to obtain a single merged convolution branch, specifically including: For each convolution branch in the neural network model, determining the corresponding convolution kernel size and bias term; Aligning the convolution kernel sizes and summing them to obtain an equivalent convolution kernel; and summing the bias terms to obtain an equivalent bias term; Based on the equivalent convolution kernel and the equivalent bias term, a single merged convolution branch is obtained.

5. The lightweight image restoration method based on multi-scale features according to claim 4 is characterized in that: Align the convolution kernel sizes and sum them to obtain an equivalent convolution kernel, specifically including: Aligning the convolution kernel sizes to obtain aligned convolution kernels; According to the convolution kernel sizes before alignment, the convolution branches are grouped so that the maximum difference in the convolution kernel sizes in each group is lower than a preset difference; For each group, the aligned convolution kernels are summed to obtain the sub-equivalent convolution kernel; For each group, perform external summation on each sub-equivalent convolution kernel to obtain the equivalent convolution kernel.

6. A lightweight image restoration device based on multi-scale features, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the lightweight image restoration method based on multi-scale features as described in any one of claims 1 to 5.

7. A non-volatile computer storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are configured to implement the lightweight image restoration method based on multi-scale features as described in any one of claims 1 to 5.

Images