Method, device and equipment for super-high-definition image restoration based on clustering center feature scanning

Abstract

The application relates to an ultra-high-definition image restoration method, device and equipment based on a clustering center feature scanning. The method comprises the following steps: constructing a clustering center scanning unit and a space-channel feature modulator, the clustering center scanning unit constructs a plurality of center features through sparse sampling of an input feature map in a feature aggregation stage, calculates a cosine similarity matrix of pixels and the center features, filters similar pixels through an activation function and aggregates to generate a clustering core feature, and then establishes an association mapping of core and non-core features by using the similarity matrix through a score diffusion stage, transfers a global modeling weight, the space-channel feature modulator extracts global features and texture detail features in parallel attention branches, fuses the two modules based on an asymmetric encoder-decoder architecture, constructs an ultra-high-definition image restoration network, restores a low-quality ultra-high-definition image after training, and outputs a high-quality image. The method can effectively solve the existing technology's display memory bottleneck and detail blur problem.

Description

Technical Field

[0001] This application relates to the field of optical image enhancement technology, and in particular to an ultra-high-definition image restoration method, apparatus and device based on cluster center feature scanning. Background Technology

[0002] With the popularization of mobile devices and streaming media technology, ultra-high-definition images have become the mainstream paradigm for visual media consumption. However, hardware limitations (such as small sensors in smartphones) and environmental degradation (such as low light, motion blur, and rain and fog interference) often lead to damage to the quality of ultra-high-definition images. There is an urgent need for efficient restoration technology to restore the structural clarity and realism of images.

[0003] Current methods for restoring low-resolution images often face memory overflow issues when dealing with ultra-high-resolution images. For example, traditional Transformer-based methods suffer from input-size-dependent quadratic complexity due to their self-attention mechanism, resulting in memory consumption of ≥24GB when processing ultra-high-resolution images, far exceeding the capacity of consumer-grade GPUs (≤12GB VRAM). Even with state-space models (SSMs, such as Mamba) with linear complexity, their pixel-level scanning mechanism still requires traversing all pixels in the image, leading to a sharp increase in complexity during hidden state propagation, making them difficult to deploy on consumer-grade hardware. Existing methods specifically designed for ultra-high-resolution images often employ high-scale downsampling (e.g., 4x downsampling) or model only low-frequency features to balance efficiency and quality. While this reduces computational cost, it results in the loss of high-frequency texture details and global semantic consistency, leading to problems such as texture blurring and structural breaks in the restored image. Summary of the Invention

[0004] Therefore, it is necessary to provide an ultra-high-definition image restoration method, apparatus, and device based on cluster center feature scanning that can achieve "low complexity, high fidelity, and consumer-grade deployment" in response to the above-mentioned technical problems.

[0005] A method for restoring ultra-high-definition images based on cluster center feature scanning, the method comprising:

[0006] A cluster center scanning unit is constructed, comprising a feature aggregation stage and a score diffusion stage. In the feature aggregation stage, the input feature map is sparsely sampled to construct multiple center features, and the cosine similarity matrix between pixel features and each of the center features is calculated. Pixels similar to each of the center features are filtered using an activation function and aggregated to generate cluster core features. In the score diffusion stage, the cosine similarity matrix is ​​used to establish the association mapping between the cluster core features and non-core features. The global modeling weights of the cluster core features are allocated and transferred to the non-core features based on the association mapping to obtain the restored features.

[0007] The spatial-channel feature modulator extracts global features and texture detail features of an input feature map through parallel channel attention branches and spatial attention branches respectively, so as to obtain a completed feature;

[0008] According to the cluster center scanning unit and the spatial-channel feature modulator, an ultra-high definition image restoration network is constructed based on an asymmetrically designed encoder and decoder architecture, and the ultra-high definition image restoration network is trained, wherein the cluster center scanning unit and the spatial-channel feature modulator are arranged in parallel in the decoder;

[0009] An ultra-high definition low-quality image to be restored is obtained, and the ultra-high definition image restoration network is used to restore the ultra-high definition low-quality image to obtain an ultra-high definition high-quality restored image.

[0010] In one embodiment, in the cluster center scanning unit:

[0011] The layer-normalized input feature map is sequentially subjected to feature transformation through a multi-layer perceptron and a depth convolutional layer, and then subjected to processing through an activation function to obtain a first intermediate feature.

[0012] After the feature aggregation is performed on the first intermediate feature, a selective scanning mechanism of Mamba is processed, and then score diffusion is performed, and the feature obtained after the score diffusion is subjected to processing through a normalization layer to obtain a second intermediate feature.

[0013] Meanwhile, after the layer-normalized input feature map is subjected to weight adjustment through a multi-layer perceptron and an activation function, element-level multiplication is performed with the second intermediate feature to obtain the restored feature.

[0014] In one embodiment, in the feature aggregation stage:

[0015] The input feature map includes a channel dimension and a spatial dimension, initial points are uniformly selected in the input feature map, and a plurality of center features are constructed by calculating the k-nearest neighbor mean values of the initial points.

[0016] The pixel features in the input feature map are mapped to a deep feature space by a multi-layer perceptron to obtain deep pixel features, and the cosine similarity between the deep pixel features and the center features is calculated.

[0017] The center features in the latent feature space are introduced, and the cluster core features are obtained according to the similarity in the cosine similarity.

[0018] In one embodiment, in the score diffusion stage:

[0019] generating global weights according to the cluster core features by a selective scanning mechanism of Mamba;

[0020] For each deep pixel feature in the deep feature space, the global weights of the plurality of cluster core features are proportionally fused according to the corresponding similarity in the cosine similarity matrix, and each deep pixel feature is updated.

[0021] In one embodiment, in the space-channel feature modulator:

[0022] The layer-normalized input feature map is input into a parallel channel attention branch and a spatial attention branch respectively to obtain channel attention weights and spatial attention weights.

[0023] Meanwhile, the layer-normalized input feature map is respectively extracted by a convolution layer and a ReLU activation function, and the extracted feature map is respectively weighted by the channel attention weights and the spatial attention weights.

[0024] The outputs of the two branches are fused by element-level addition to obtain the completed feature.

[0025] In one embodiment, in the decoder:

[0026] The recovered feature and the completed feature output by the cluster center scanning unit and the space-channel feature modulator are element-wise added to obtain a feature, and the output feature of the decoder is obtained by a feedforward network and upsampling.

[0027] In one embodiment, in the super-high-definition image restoration network:

[0028] The encoder only includes a feedforward network;

[0029] A bottleneck layer is further arranged between the encoder and the decoder, the bottleneck layer performs deep feature extraction on the encoder output feature, and the extracted feature is channel-wise spliced with the feature output by the encoder and then input into the decoder.

[0030] The feature output by the decoder is sequentially subjected to upsampling, convolution layer, and degraded image superposition through a feedforward network to obtain a restored image.

[0031] In one embodiment, before the super-high-definition low-quality image is input into the super-high-definition image restoration network, a random cropping and flipping data augmentation operation is performed on the super-high-definition low-quality image.

[0032] The application also provides a super-high-definition image restoration device based on cluster center feature scanning, the device comprising:

[0033] The clustering center scanning module is configured to construct a clustering center scanning unit, which includes a feature aggregation stage and a score diffusion stage. In the feature aggregation stage, the input feature map is sparsely sampled, a plurality of center features are constructed, and a cosine similarity matrix of pixel features and each center feature is calculated. An activation function is used to filter and aggregate the pixels similar to each center feature to generate a clustering core feature. In the score diffusion stage, the cosine similarity matrix is used to establish an association mapping between the clustering core feature and non-core features. The global modeling weight of the clustering core feature is distributed and transferred to the non-core features based on the association mapping to obtain a restored feature.

[0034] The space-channel feature modulator construction module is configured to construct a space-channel feature modulator. The space-channel feature modulator extracts global features and texture detail features of an input feature map through parallel channel attention branches and spatial attention branches, thereby obtaining a completed feature.

[0035] The super-high-definition image restoration network construction module is configured to construct a super-high-definition image restoration network based on an asymmetrically designed encoder and decoder architecture according to the clustering center scanning unit and the space-channel feature modulator, and train the super-high-definition image restoration network. The clustering center scanning unit and the space-channel feature modulator are arranged in parallel in the decoder.

[0036] The super-high-definition low-quality image restoration module is configured to obtain a super-high-definition low-quality image to be restored, restore the super-high-definition low-quality image using the trained super-high-definition image restoration network, and obtain a super-high-definition high-quality restored image.

[0037] A computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps of the above-described super-high-definition image restoration method based on clustering center feature scanning are implemented.

[0038] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the above-described super-high-definition image restoration method based on clustering center feature scanning are implemented.

[0039] The above-mentioned super-high-definition image restoration method, device and equipment based on cluster center feature scanning, by constructing a cluster center scanning unit and a space-channel feature modulator, in the cluster center scanning unit, by sparse sampling the input feature map to construct a plurality of center features in the feature aggregation stage, and calculating the cosine similarity matrix of the pixel features and each center feature, using an activation function to filter and aggregate the pixels similar to each center feature to generate a cluster core feature, and by using the cosine similarity matrix to establish the association mapping between the cluster core feature and the non-core feature in the score diffusion stage, the global modeling weight of the cluster core feature is distributed and transferred to the non-core feature based on the association mapping to obtain the restored feature, in the space-channel feature modulator, the global feature and the texture detail feature of the input feature map are extracted by the parallel channel attention branch and the spatial attention branch respectively to obtain the completion feature, according to the cluster center scanning unit and the space-channel feature modulator, the encoder and decoder architecture based on asymmetric design is constructed, the trained super-high-definition image restoration network is used to restore the super-high-definition low-quality image, and the super-high-definition high-quality restored image is obtained. The method can effectively solve the existing method memory bottleneck and detail blur problem, can provide high-fidelity super-high-definition image support for automatic driving, security monitoring, remote sensing mapping and other fields, and has significant practical value and technical innovation. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 FIG. 1 is a flowchart of a super-high-definition image restoration method based on cluster center feature scanning in one embodiment;

[0041] Figure 2 FIG. 2 is a structural diagram of a super-high-definition image restoration network in one embodiment, wherein Figure 2 (a) represents the overall structure of the super-high-definition image restoration network, Figure 2 (b) represents the structure of the cluster center scanning unit, Figure 2 (c) represents the structure of the space-channel feature modulator;

[0042] Figure 3 FIG. 3 is a structural block diagram of a super-high-definition image restoration device based on cluster center feature scanning in one embodiment;

[0043] Figure 4 FIG. 4 is an internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0045] To overcome the shortcomings of the prior art, an ultra-high-definition image restoration method based on cluster center feature scanning is proposed in the method, which aims to innovate the traditional visual state space model pixel scanning mechanism, model only the core pixel points, and spread the global information from sparse points to the whole area through a unique fractional diffusion mechanism, to realize high-quality restoration of ultra-high-definition images while maintaining linear complexity. Specifically, a dual-path fusion architecture is constructed in the method, in which the cluster center scanning unit realizes efficient global dependence modeling through feature aggregation and fractional diffusion, and the space-channel feature modulator compensates for local details through an attention mechanism, finally realizing end-to-end ultra-high-definition image restoration in an encoder-decoder framework. Experiments show that in multiple ultra-high-definition restoration tasks, the method is superior to the existing optimal method in terms of quantitative indicators such as PSNR and SSIM, and realizes full-resolution inference on a consumer-grade GPU.

[0046] In one embodiment, as shown in Figure 1 The ultra-high-definition image restoration method based on cluster center feature scanning specifically includes the following steps:

[0047] Step S100, a cluster center scanning unit is constructed, which includes a feature aggregation stage and a fractional diffusion stage. In the feature aggregation stage, the input feature map is sparsely sampled to construct multiple center features, and a cosine similarity matrix of pixel features and each center feature is calculated. The similar pixels to each center feature are selected and aggregated using an activation function to generate a cluster core feature. In the fractional diffusion stage, the association mapping between the cluster core feature and the non-core feature is established using the cosine similarity matrix. The global modeling weight of the cluster core feature is distributed and transferred to the non-core feature based on the association mapping to obtain the restoration feature.

[0048] Step S110, a space-channel feature modulator is constructed, which extracts global features and texture detail features of the input feature map through parallel channel attention branches and spatial attention branches, respectively, to obtain a complete feature.

[0049] Step S120, based on the cluster center scanning unit and the space-channel feature modulator, an ultra-high-definition image restoration network is constructed based on an asymmetrically designed encoder and decoder architecture, and the ultra-high-definition image restoration network is trained, wherein the cluster center scanning unit and the space-channel feature modulator are arranged in parallel in the decoder.

[0050] Step S130, an ultra-high-definition low-quality image to be restored is obtained, and the trained ultra-high-definition image restoration network is used to restore the ultra-high-definition low-quality image to obtain an ultra-high-definition high-quality restored image.

[0051] Since adjacent regions in visual image space tend to converge to similar feature weight patterns, they naturally contain a high degree of semantic redundancy. To this end, using global modeling of visual state space for scanning, pixel by pixel results in an extremely long sequence with a high degree of redundancy, which causes a computational bottleneck. In step S100, the cluster center scanning unit is constructed to reduce the huge pixel set to a number of representative cluster center features far less than the total number of pixels by dynamic clustering. The subsequent complex state space model (block) will only act on these center features rather than all pixels, thereby fundamentally reducing the computational and memory overhead.

[0052] In this embodiment, in the cluster center scanning unit (CCSM): the layer-normalized input feature map is sequentially transformed by a multi-layer perceptron and a deep separable convolution layer, and then processed by an activation function to obtain a first intermediate feature. After performing feature aggregation on the intermediate feature, the selective scanning mechanism of Mamba is processed, and then score diffusion is performed. The feature obtained after score diffusion is processed by a normalization layer to obtain a second intermediate feature. At the same time, the layer-normalized input feature map is adjusted by a multi-layer perceptron and an activation function, and then multiplied element-wise with the second intermediate feature to obtain a restored feature.

[0053] Specifically, for the layer-normalized input feature map , first, it is transformed by a multi-layer perceptron (MLP) and a deep separable convolution (DWConv), and then processed by a SiLU activation function. The process is represented as:

[0054]

[0055] Further, the feature aggregation (FA) and score diffusion (SD) operations are performed, combined with the selective scanning mechanism (S6) of Mamba and the normalization layer (Norm) processing. The process is represented as:

[0056]

[0057] Next, the input feature map is adjusted by a SiLU-activated MLP, and multiplied element-wise with the feature map to obtain the CCSM output :

[0058] .

[0059] In the feature aggregation stage, the input feature map contains channel dimension and spatial dimension, initial points are selected uniformly in the input feature map, a plurality of center features are constructed by calculating the k-nearest neighbor mean of each initial point, then the pixel features in the input feature map are mapped to deep layer feature space by using layer perception to obtain deep layer pixel features, and the cosine similarity between the deep layer pixel features and the center features in the latent feature space is calculated, the center features in the latent feature space are introduced, and the clustering core features are obtained according to the similarity in the cosine similarity.

[0060] Specifically, let the input feature map (C is the number of channels, ( W) is the spatial dimension), by uniformly selecting initial points and calculating the k-nearest neighbor mean of each initial point , a plurality of center features are constructed The clustering centers are initialized, and here the centers are not directly randomly initialized. This method utilizes the local consistency of image features, so that the initial centers can better represent the local structure of the image, and compared with random initialization, it can accelerate the convergence of the clustering process and improve the stability. Then, the pixel features are mapped to a specific value space to obtain , the cosine similarity between the points and the centers is calculated using and , and the expression is:

[0061]

[0062] Next, the centers in the latent feature space are introduced (the calculation method is the same as ), for a clustering cluster containing data points, the clustering core features are generated by aggregation through the following formula:

[0063] ;

[0064] ;

[0065] In the above formula, is a normalization factor, is a Sigmoid activation function, is the similarity of the pixel feature and the clustering center, is the clustering point feature mapped to the value space by , and are learnable similarity adjustment parameters. For each cluster, the core feature is obtained by weighted average of all pixel features, and the weight is determined by the similarity of them and the center.​

[0066] In this embodiment, while the feature aggregation stage efficiently achieves long-distance dependency modeling of ultra-high-definition images, this comes at the cost of some performance. Therefore, this method uses coarse-grained scanning instead of the previous full-pixel scanning. To address this, and to solve the problems of information loss and dimensionality mismatch, a fractional diffusion mechanism is constructed to efficiently and reasonably "propagate" or "difflate" the learned global dependencies of the center points to every pixel, thereby achieving global perception reconstruction from sparse points to dense regions.

[0067] Specifically, in the score diffusion stage, global weights are generated based on the cluster core features through Mamba's selective scanning mechanism. For each deep pixel feature in the deep feature space, the global weights of multiple cluster core features are proportionally fused with the corresponding similarity in the cosine similarity matrix to update each deep pixel feature.

[0068] Clustering core features obtained from feature aggregation Global weights are generated through Mamba's selective scanning mechanism (S6), the process of which is as follows:

[0069]

[0070] In the above formula, S6 is a selective scanning mechanism that can dynamically determine how to propagate and forget information based on the input (i.e., these central features), thereby effectively capturing long-range dependencies.

[0071] Furthermore, the cosine similarity matrix already calculated during the feature aggregation stage is utilized. As a blueprint for allocation. For each pixel in the deep feature space. Its update formula is:

[0072]

[0073] In the above formula, It is the Sigmoid activation function. Used to perform feature transformation on the weighted fusion results, ensuring that non-core pixels can acquire multi-source global information.

[0074] In step S110, an efficient attention mechanism is used to recalibrate the feature response in both spatial and channel dimensions, thereby amplifying important detailed features and suppressing irrelevant information. This serves as a supplementary information source to compensate for information loss due to non-full-pixel scanning in the Cluster Center Scanning Unit (CCSM). The efficient Spatial-Channel Feature Modulator (SCFM) employs a dual-branch parallel structure to process spatial and channel information separately, and finally fuses the results through element-wise addition. Its input is the same normalized feature map as that of the CCSM. .

[0075] In this embodiment, in the spatial-channel feature modulator, the layer-normalized input feature map is input into the parallel channel attention branch and spatial attention branch respectively to obtain channel attention weights and spatial attention weights. At the same time, the layer-normalized input feature map is processed by convolutional layer and ReLU activation function for feature extraction. Then, the extracted feature map is weighted by channel attention weight and spatial attention weight respectively. The outputs of the two branches are fused by element-level addition to obtain the completed feature.

[0076] Specifically, the process in the spatial-channel feature modulator can be represented as follows:

[0077] ;

[0078] ;

[0079] ;

[0080] ;

[0081] In the above formula, This represents the convolution operation. For activation function, The input feature map is after layer normalization. and These are channel attention weights and spatial attention weights, respectively, obtained through adaptive learning via channel compression and spatial compression mechanisms, used to highlight key channel and spatial region features. This refers to the output characteristics of a spatial-channel feature modulator.

[0082] In step S120, the core objective is to achieve high-quality restoration of UHD images on consumer-grade hardware. This requires the network architecture to achieve a delicate balance between computational efficiency and model performance. In this method, directly applying a standard symmetric encoder-decoder structure (such as the U-Net architecture) would result in huge computational overhead when extracting high-resolution features at a deep level in the encoder. Therefore, this method designs an asymmetric encoder-decoder architecture, the core idea of ​​which is to make the encoder lightweight for efficient dimensionality reduction and primary feature extraction, while concentrating the complex global modeling and detail compensation tasks in the decoder.

[0083] In this embodiment, the structure of the ultra-high-definition image restoration network is as follows: Figure 2As shown, the encoder contains only a feedforward network. A bottleneck layer is placed between the encoder and decoder. The bottleneck layer performs deep feature extraction on the encoder output features. The extracted features are then concatenated with the encoder output features at the channel level and input into the decoder. The decoder output features pass through the feedforward network, and then pass through upsampling and convolutional layers in sequence before being superimposed on the degraded image to obtain the restored image.

[0084] In this embodiment, in the decoder, the restored features and the completed features output by the cluster center scanning unit and the spatial-channel feature modulator are added element by element to obtain the features, and the decoder output features are obtained through a feedforward network and upsampling.

[0085] Specifically, the encoder contains only a feedforward neural network (FFN) to reduce computational load, while the decoder integrates a cluster center scanning unit (CCSM), a spatial-channel feature modulator (SCFM), and the FFN. The network includes... Each layer, both encoder and decoder adopt The basic block structure. The bottleneck layer and the refinement stage each contain... Each block has a basic embedding dimension of 32. Encoder features are injected into the decoder via skip connections. Finally, the learned residuals are added to the degraded input image to obtain the final restored image.

[0086] In step S130, before inputting the ultra-high-definition low-quality image into the ultra-high-definition image restoration network, the ultra-high-definition low-quality image is subjected to random cropping and flipping data augmentation operations.

[0087] In this paper, the effectiveness of the proposed method is also demonstrated through experiments, and the quantitative experimental results are shown in Tables 1 and 2.

[0088] Table 1. Quantitative experimental results of each method on the UHD-Blur test dataset.

[0089]

[0090] Table 2. Quantitative experimental results of each method on the 4KRain-13k test dataset.

[0091]

[0092] The aforementioned ultra-high-definition image restoration method based on cluster center feature scanning addresses the problems of high computational complexity, large memory consumption, and easy loss of global context and high-frequency details in existing ultra-high-definition image restoration technologies on consumer-grade hardware. It constructs an encoder-decoder asymmetric architecture and a dual-module collaborative optimization framework to achieve efficient and high-fidelity ultra-high-definition image restoration. First, a cluster center scanning unit (CCSM) is designed, innovating the traditional full-pixel scanning mechanism through a joint process of feature aggregation and fractional diffusion: in the feature aggregation stage, cluster centers are dynamically sampled based on pixel feature similarity, and global modeling is performed only on core features to reduce computational complexity; in the fractional diffusion stage, the global weights of core pixels are propagated to non-core pixels based on the similarity matrix, achieving region-level restoration from sparse points to complete regions, while avoiding overfitting caused by dependence on a single cluster. Second, a spatial-channel feature modulator (SCFM) is configured, which performs feature capture in full-resolution image convolution through parallel channel attention and spatial attention branches, compensating for the accuracy loss of cluster scanning and balancing feature extraction efficiency and detail preservation. Finally, by integrating only a feedforward network (FFN) into the encoder and then incorporating the aforementioned modules into the decoder, this asymmetric encoder-decoder architecture further reduces computational load and memory usage while ensuring image restoration quality. Experimental results show that this method achieves the best performance compared to existing methods in ultra-high-definition image deblurring and deraining tasks. It can effectively solve the memory bottleneck and detail blurring problems of existing methods, and can provide high-fidelity ultra-high-definition image support for fields such as autonomous driving, security monitoring, and remote sensing mapping, demonstrating significant practical value and technological innovation.

[0093] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0094] In one embodiment, such as Figure 3 As shown, an ultra-high-definition image restoration device based on cluster center feature scanning is provided, including: a cluster center scanning construction module 200, a spatial-channel feature modulator construction module 210, an ultra-high-definition image restoration network construction module 220, and an ultra-high-definition low-quality image restoration module 230, wherein:

[0095] The cluster center scanning construction module 200 is used to construct a cluster center scanning unit. The cluster center scanning unit includes a feature aggregation stage and a score diffusion stage. In the feature aggregation stage, the input feature map is sparsely sampled to construct multiple center features, and the cosine similarity matrix between pixel features and each of the center features is calculated. Pixels similar to each of the center features are filtered using an activation function and aggregated to generate cluster core features. In the score diffusion stage, the cosine similarity matrix is ​​used to establish the association mapping between the cluster core features and non-core features. The global modeling weights of the cluster core features are allocated and transferred to the non-core features based on the association mapping to obtain the restored features.

[0096] The spatial-channel feature modulator construction module 210 is used to construct a spatial-channel feature modulator. The spatial-channel feature modulator extracts global features and texture detail features of the input feature map through parallel channel attention branches and spatial attention branches, respectively, thereby obtaining complete features.

[0097] The ultra-high-definition image restoration network construction module 220 constructs an ultra-high-definition image restoration network based on the cluster center scanning unit and the spatial-channel feature modulator, and trains the ultra-high-definition image restoration network. The cluster center scanning unit and the spatial-channel feature modulator are arranged in parallel in the decoder.

[0098] The ultra-high-definition low-quality image restoration module 230 is used to acquire the ultra-high-definition low-quality image to be restored, and use the trained ultra-high-definition image restoration network to restore the ultra-high-definition low-quality image to obtain an ultra-high-definition high-quality restored image.

[0099] Specific limitations regarding the ultra-high-definition image restoration device based on cluster center feature scanning can be found in the limitations of the ultra-high-definition image restoration method based on cluster center feature scanning mentioned above, and will not be repeated here. Each module in the aforementioned ultra-high-definition image restoration device based on cluster center feature scanning can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0100] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements an ultra-high-definition image restoration method based on cluster center feature scanning. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0101] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0102] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0103] A cluster center scanning unit is constructed, comprising a feature aggregation stage and a score diffusion stage. In the feature aggregation stage, the input feature map is sparsely sampled to construct multiple center features, and the cosine similarity matrix between pixel features and each of the center features is calculated. Pixels similar to each of the center features are filtered using an activation function and aggregated to generate cluster core features. In the score diffusion stage, the cosine similarity matrix is ​​used to establish the association mapping between the cluster core features and non-core features. The global modeling weights of the cluster core features are allocated and transferred to the non-core features based on the association mapping to obtain the restored features.

[0104] A spatial-channel feature modulator is constructed, which extracts global features and texture detail features of the input feature map through parallel channel attention branches and spatial attention branches, respectively, thereby obtaining the complete features;

[0105] Based on the cluster center scanning unit and the spatial-channel feature modulator, an ultra-high-definition image restoration network is constructed using an encoder and decoder architecture with an asymmetric design, and the ultra-high-definition image restoration network is trained. The cluster center scanning unit and the spatial-channel feature modulator are arranged in parallel in the decoder.

[0106] A high-definition low-quality image to be restored is acquired, and the trained high-definition image restoration network is used to restore the high-definition low-quality image to obtain a high-definition high-quality restored image.

[0107] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0108] A cluster center scanning unit is constructed, comprising a feature aggregation stage and a score diffusion stage. In the feature aggregation stage, the input feature map is sparsely sampled to construct multiple center features, and the cosine similarity matrix between pixel features and each of the center features is calculated. Pixels similar to each of the center features are filtered using an activation function and aggregated to generate cluster core features. In the score diffusion stage, the cosine similarity matrix is ​​used to establish the association mapping between the cluster core features and non-core features. The global modeling weights of the cluster core features are allocated and transferred to the non-core features based on the association mapping to obtain the restored features.

[0109] A spatial-channel feature modulator is constructed, which extracts global features and texture detail features of the input feature map through parallel channel attention branches and spatial attention branches, respectively, thereby obtaining the complete features;

[0110] Based on the cluster center scanning unit and the spatial-channel feature modulator, an ultra-high-definition image restoration network is constructed using an encoder and decoder architecture with an asymmetric design, and the ultra-high-definition image restoration network is trained. The cluster center scanning unit and the spatial-channel feature modulator are arranged in parallel in the decoder.

[0111] A high-definition low-quality image to be restored is acquired, and the trained high-definition image restoration network is used to restore the high-definition low-quality image to obtain a high-definition high-quality restored image.

[0112] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0113] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for restoring ultra-high-definition images based on cluster center feature scanning, characterized in that, The method includes: A cluster center scanning unit is constructed, comprising a feature aggregation stage and a score diffusion stage. In the feature aggregation stage, the input feature map is sparsely sampled to construct multiple center features, and the cosine similarity matrix between pixel features and each of the center features is calculated. Pixels similar to each of the center features are filtered using an activation function and aggregated to generate cluster core features. In the score diffusion stage, the cosine similarity matrix is ​​used to establish the association mapping between the cluster core features and non-core features. The global modeling weights of the cluster core features are allocated and transferred to the non-core features based on the association mapping to obtain the restored features. A spatial-channel feature modulator is constructed, which extracts global features and texture detail features of the input feature map through parallel channel attention branches and spatial attention branches, respectively, thereby obtaining the complete features; Based on the cluster center scanning unit and the spatial-channel feature modulator, an ultra-high-definition image restoration network is constructed using an asymmetric encoder and decoder architecture. The ultra-high-definition image restoration network is then trained. The cluster center scanning unit and the spatial-channel feature modulator are arranged in parallel in the decoder. In the ultra-high-definition image restoration network, the encoder contains only a feedforward network. A bottleneck layer is also set between the encoder and the decoder. The bottleneck layer performs deep feature extraction on the output features of the encoder. The extracted features are concatenated with the output features of the encoder at the channel level and then input into the decoder. The output features of the decoder pass through the feedforward network and are then sequentially passed through upsampling and convolutional layers and superimposed on the degraded image to obtain the restored image. A high-definition low-quality image to be restored is acquired, and the trained high-definition image restoration network is used to restore the high-definition low-quality image to obtain a high-definition high-quality restored image.

2. The ultra-high-definition image restoration method based on cluster center feature scanning according to claim 1, characterized in that, In the cluster center scanning unit: The input feature map after layer normalization is passed through a multilayer perceptron and a deep convolutional layer for feature transformation, and then the activation function is used to process the input feature map after feature transformation to obtain the first intermediate feature. After performing feature aggregation on the first intermediate feature, Mamba's selective scanning mechanism is applied, followed by fractional diffusion. The features obtained after fractional diffusion are then processed by a normalization layer to obtain the second intermediate feature. Meanwhile, after adjusting the weights of the input feature map after layer normalization using a multilayer perceptron and activation function, element-wise multiplication is performed with the second intermediate feature to obtain the restored feature.

3. The ultra-high-definition image restoration method based on cluster center feature scanning according to claim 2, characterized in that, During the feature aggregation stage: The input feature map includes channel dimension and spatial dimension. Initial points are uniformly selected in the input feature map, and multiple central features are constructed by calculating the k nearest neighbor mean of each initial point. The pixel features in the input feature map are mapped to the deep feature space using a multilayer perceptron to obtain deep pixel features. Then, the cosine similarity between the deep pixel features and the central features is calculated. The central features in the latent feature space are introduced, and the clustering core features are obtained based on the similarity index in the cosine similarity.

4. The ultra-high-definition image restoration method based on cluster center feature scanning according to claim 3, characterized in that, In the fractional diffusion stage: Global weights are generated based on the core clustering features using Mamba's selective scanning mechanism; For each deep pixel feature in the deep feature space, the global weights of multiple clustering core features are proportionally fused together with the corresponding similarity in the cosine similarity matrix to update each deep pixel feature.

5. The ultra-high-definition image restoration method based on cluster center feature scanning according to claim 4, characterized in that, In the spatial-channel feature modulator: The input feature map after layer normalization is fed into the parallel channel attention branch and spatial attention branch respectively to obtain the channel attention weight and spatial attention weight; Meanwhile, the input feature maps after layer normalization are processed by convolutional layers and ReLU activation functions to extract features, and then the extracted feature maps are weighted by the channel attention weights and spatial attention weights respectively. The completion feature is obtained by fusing the outputs of the two branches through element-level addition.

6. The ultra-high-definition image restoration method based on cluster center feature scanning according to claim 5, characterized in that, In the decoder: The restored and completed features output by the cluster center scanning unit and the spatial-channel feature modulator are element-wise summed to obtain the feature, which is then passed through a feedforward network and upsampling to obtain the output feature of the decoder.

7. The ultra-high-definition image restoration method based on cluster center feature scanning according to claim 6, characterized in that, Before inputting the ultra-high-definition low-quality image into the ultra-high-definition image restoration network, the ultra-high-definition low-quality image is subjected to random cropping and flipping data augmentation operations.

8. A device for restoring ultra-high-definition images based on cluster center feature scanning, characterized in that, The device includes: A cluster center scanning construction module is used to construct a cluster center scanning unit. The cluster center scanning unit includes a feature aggregation stage and a score diffusion stage. In the feature aggregation stage, the input feature map is sparsely sampled to construct multiple center features, and the cosine similarity matrix between pixel features and each of the center features is calculated. Pixels similar to each of the center features are filtered using an activation function and aggregated to generate cluster core features. In the score diffusion stage, the cosine similarity matrix is ​​used to establish the association mapping between the cluster core features and non-core features. The global modeling weights of the cluster core features are allocated and transferred to the non-core features based on the association mapping to obtain the restored features. A spatial-channel feature modulator construction module is used to construct a spatial-channel feature modulator. The spatial-channel feature modulator extracts global features and texture detail features of the input feature map through parallel channel attention branches and spatial attention branches, respectively, thereby obtaining complete features. An ultra-high-definition image restoration network construction module constructs an ultra-high-definition image restoration network based on the cluster center scanning unit and the spatial-channel feature modulator, using an asymmetric encoder and decoder architecture. The module then trains the ultra-high-definition image restoration network. Specifically, the cluster center scanning unit and the spatial-channel feature modulator are arranged in parallel within the decoder. In the ultra-high-definition image restoration network, the encoder contains only a feedforward network, and a bottleneck layer is provided between the encoder and decoder. This bottleneck layer performs deep feature extraction on the encoder output features. The extracted features are then concatenated with the encoder output features at the channel level and input into the decoder. The decoder output features pass through the feedforward network, then sequentially pass through upsampling and convolutional layers before being superimposed on the degraded image to obtain the restored image. The ultra-high-definition low-quality image restoration module is used to acquire the ultra-high-definition low-quality image to be restored, and to restore the ultra-high-definition low-quality image using a trained ultra-high-definition image restoration network to obtain an ultra-high-definition high-quality restored image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images