Feature clustering and region-adaptive multi-discriminator generative adversarial network enhancement method

Abstract

The present invention relates to the field of image processing technology, and in particular, provides a method for enhancing a multi-discriminator generative adversarial network with feature clustering and regional adaptation. The method comprises inputting a low-resolution image into a generator to generate a super-resolution image; inputting both the super-resolution image and a high-resolution real image into a multi-discriminator for discrimination, and weightedly fusing the outputs of each discriminator using weights to obtain a comprehensive discrimination result; calculating the loss using a loss function based on the comprehensive discrimination result to obtain a loss value; updating parameters based on the loss value and a gradient descent algorithm to optimize the multi-discriminator generative adversarial network; repeating the above steps until the iteration is complete or the loss value is minimized, obtaining optimal weights, and applying them to the multi-discriminator generative adversarial network. At this point, the low-resolution image is input into the generator to obtain the final super-resolution image. This method addresses the limitations of a single discriminator in traditional models and improves the generalization and stability of the network.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a Feature Clustering and Region-Adaptive Multi-Discriminator Generative Adversarial Network Enhancement Method (FCR-ESRGAN). Background Art

[0002] Image super-resolution (SISR) technology has been widely used in various fields, including medicine, remote sensing imaging, and video surveillance. By restoring details in low-resolution images, it increases image clarity and readability. However, due to the limited learning capabilities of current super-resolution technologies, the restored high-resolution images are often accompanied by noise and artifacts.

[0003] In existing technologies, image super-resolution mainly relies on interpolation and regularization methods, such as bicubic interpolation and bilinear interpolation, to restore image details by introducing priors and constraints. In recent years, deep learning-based methods, such as convolutional neural networks (CNNs) and generative adversarial networks (GANs), have become more effective solutions. The super-resolution generative adversarial network (SRGAN) significantly improves visual quality through perceptual loss; the image restoration model SwinIR combines the advantages of CNN and Transformer to improve image restoration efficiency. However, the traditional single discriminator GAN architecture has limitations when processing images with complex textures, such as a limited feature extraction range, a single optimization direction, difficulty in balancing the generation quality of multimodal features, and an inability to effectively integrate cross-scale features, resulting in artifacts or over-smoothing problems when reconstructing image details. Summary of the Invention

[0004] In view of this, the present invention provides a feature clustering and region-adaptive multi-discriminator generative adversarial network enhancement method to solve the limitations of the single discriminator in the traditional model and improve the generalization and stability of the network.

[0005] In a first aspect, the present invention provides a method for enhancing a multi-discriminator generative adversarial network with feature clustering and region adaptation, the method comprising:

[0006] S1, low resolution image Input to the generator to generate super-resolution images ;

[0007] S2, super-resolution image and high-resolution real images The multiple discriminators are input together for discrimination, and the outputs of each discriminator are weighted and fused to obtain a comprehensive discrimination result;

[0008] S3. Based on the comprehensive discrimination results, use the loss function to calculate the loss and obtain the loss value;

[0009] S4. Update parameters based on the loss value and gradient descent algorithm to optimize the multi-discriminator generative adversarial network;

[0010] S5. Repeat the above steps until the iteration is completed or the loss value reaches the minimum, and the optimal weight is obtained, and it is applied to the multi-discriminator generative adversarial network. At this time, the low-resolution image Input to the generator to obtain the final super-resolution image.

[0011] Optionally, the S1 includes:

[0012] Low-resolution images The input is fed into the generator, where the initial features are first extracted through the convolutional layer. Then, deep features are extracted through multiple basic blocks. Skip connections are then used to effectively transfer shallow layer information to deep layers. Upsampling is performed at the back end of the multi-discriminator generative adversarial network to restore high-resolution features, and details are optimized through multiple convolutional layers, ultimately outputting a super-resolution image. .

[0013] Optionally, the S2 includes:

[0014] The single discriminator of the generative adversarial network ESRGAN is improved into a multi-discriminator, which includes the basic discriminator Discriminator1, the feature clustering discriminator Discriminator2, and the feature block discriminator Discriminator3; the basic discriminator is the discriminator in ESRGAN, the feature clustering discriminator includes feature clustering, routing mechanism, and expert network; the feature block discriminator includes region adaptive processing, routing mechanism, and expert network;

[0015] The basic discriminator process includes adopting a convolutional neural network (CNN)-based architecture, which consists of multiple convolutional layers and activation functions Leaky ReLU, gradually extracting image features, and using fully connected layers and activation functions Sigmoid to discriminate whether the input image is a super-resolution image. or high-resolution real images ;

[0016] The process of feature clustering discriminator and feature block discriminator includes feature extraction module FEM, spectral clustering module ISKM and image blocker ITS, which respectively perform super-resolution image classification on the input. and high-resolution real images After processing by the feature routing module FRM, the feature map is obtained. to , and through n expert sub-networks to After further processing, the comprehensive discrimination result is finally obtained through the feature selection module CFS.

[0017] Optionally, the spectral clustering module ISKM includes:

[0018] ISKM uses a container-filling-based strategy to cluster features and dynamically adjusts the center value of the cluster to make the clustering result more stable and robust. Its expression is:

[0019] ;

[0020] Each cluster As a container, used to store the assigned pixel features ,in Indicates the number of pixels filled, is the center of the cluster, During the clustering process, new pixel features are assigned to the nearest cluster based on the distance from the cluster center, and the mean is updated. When the container is not full, new pixels are added directly and the mean is updated. When the container is full, existing pixels are replaced to keep the cluster size stable.

[0021] Optionally, the process of the image blocker ITS includes:

[0022] ITS extracts local features through sliding windows and reorganizes data using extended Unfold operations to improve feature expression capabilities and adapt to subsequent clustering analysis; input feature maps The shape is ,in is the batch size, is the number of channels, is the feature map size; ITS is through the window size and step length Extract local area, if The windows do not overlap if There is overlap, which helps to enhance the ability to capture local features; then, ITS uses the expanded Unfold operation to The region is flattened to -dimensional feature vector and convert the entire feature map into shape tensor, where is the total number of windows, calculated as .

[0023] Optionally, the process of the feature routing module FRM includes:

[0024] FRM uses the routing matrix to assign the optimal expert subnetwork ESN to the feature map processed by ISKM or ITS to improve the feature expression ability and computational efficiency. First, the input feature map Through the learnable weight matrix Calculate the routing matrix R, and obtain the distribution weight of each sub-network through Softmax normalization; then, according to the maximum matching degree , each feature map The features are sent to the most suitable expert sub-network for processing; the processed features are aggregated in the final fusion stage through the feature selection module CFS to form global features; among them, the weight matrix Input into the routing mechanism to obtain the routing matrix R, which is expressed as: , NULL means invalid.

[0025] Optionally, the S3 includes:

[0026] Adopt reconstruction loss and perceptual loss to optimize low-resolution images With super-resolution images Mapping between them to improve image quality; calculate super-resolution images With high-resolution real images The perceptual loss between is used to ensure the perceptual quality of the generated image; the intra-class feature compactness loss is calculated , used to reduce the intra-class feature dispersion; calculate the inter-class feature dispersion loss , used to increase the spacing between features of different categories; calculate expert load consistency loss , used to balance the task allocation of expert networks; by integrating the previous adversarial loss , construct the discriminator loss function , whose expression is:

[0027] ;

[0028] in, 、 and is the weight coefficient; is the feature density loss, including the intra-class feature compactness loss and inter-class feature dispersion loss , whose expression is: ; and is the weight coefficient, which is used to adjust the influence of the two parts of loss.

[0029] Optionally, the intra-class feature compactness loss is used to reduce the dispersion of intra-class features so that feature points of the same category are more concentrated in the feature space. Its expression is:

[0030] ;

[0031] in, and Represents the feature vector of similar samples. The loss calculation uses the Gaussian kernel function to measure the similarity of each pair of samples; Euclidean distance The smaller it is, the closer the value of the exponential function is to 1, indicating that the samples within the class are closer;

[0032] The inter-class feature dispersion loss is used to increase the feature spacing of different categories, so that the features of different categories are distributed far away, thereby improving the separability of classification. Its expression is:

[0033] ;

[0034] in, and Represents the normalized feature vectors of different categories. The inter-class feature dispersion loss calculates the cosine similarity of feature points between all categories and takes the average value.

[0035] Optionally, the expert load consistency loss It aims to optimize the load balancing of different expert sub-networks ESN during network allocation and ensure that all expert sub-networks can evenly share the tasks during feature processing. Its expression is:

[0036] ;

[0037] in, represents the number of expert sub-networks, represents the total number of input feature blocks, Reflect the The feature blocks are assigned to The probability of an expert sub-network, As a binary indicator, it is used to indicate whether the feature block is actually assigned to the expert sub-network; the calculation method of expert load consistency loss is to first calculate the average allocation weight of each expert and actual distribution , then take their product and weight normalize them to measure the overall load balancing.

[0038] In a second aspect, an embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the feature clustering and region-adaptive multi-discriminator generative adversarial network enhancement method in the first aspect or any possible implementation of the first aspect.

[0039] In a third aspect, an embodiment of the present invention provides a method comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed, cause the device to execute the feature clustering and region-adaptive multi-discriminator generative adversarial network enhancement method in the first aspect or any possible implementation of the first aspect.

[0040] In the technical solution provided by the present invention, the method includes inputting a low-resolution image into a generator to generate a super-resolution image; inputting the super-resolution image and the high-resolution real image into a multi-discriminator for discrimination, and the outputs of each discriminator are weighted and fused respectively through weights to obtain a comprehensive discrimination result; according to the comprehensive discrimination result, the loss is calculated using a loss function to obtain a loss value; according to the loss value and the gradient descent algorithm, the parameters are updated to optimize the multi-discriminator generative adversarial network; repeating the above steps until the iteration is completed or the loss value reaches the minimum, the optimal weight is obtained, and it is applied to the multi-discriminator generative adversarial network, at this time the low-resolution image is input into the generator to obtain the final super-resolution image. This method solves the limitation problem of the single discriminator in the traditional model and improves the generalization and stability of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 A flowchart of a method for enhancing a feature clustering and region-adaptive multi-discriminator generative adversarial network provided by an embodiment of the present invention;

[0043] Figure 2 A diagram of the generator structure provided by an embodiment of the present invention;

[0044] Figure 3 A diagram of the architecture of a multi-discriminator generative adversarial network provided by an embodiment of the present invention;

[0045] Figure 4 A diagram of the discriminator architecture provided by an embodiment of the present invention;

[0046] Figure 5 An example diagram of visual comparison provided by an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0049] It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0050] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0051] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the associated objects.

[0052] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0053] Figure 1 The flowchart of the feature clustering and region-adaptive multi-discriminator generative adversarial network enhancement method provided by the embodiment of the present invention is as follows: Figure 1 As shown, the method includes:

[0054] S1, low resolution image Input to the generator to generate super-resolution images .

[0055] In this embodiment of the present invention, the generator uses the ESRGAN generator, which utilizes a Residual-in-Residual Dense Block (RRDB) architecture and removes batch normalization (BN) layers to preserve image details and reduce unnecessary computational overhead. RRDB, composed of multiple Dense Residual Blocks (DRBs), enhances feature extraction and reuse while mitigating the vanishing gradient problem. The generator performs upsampling through sub-pixel convolution or deconvolution to achieve super-resolution reconstruction. This overall architecture, combined with deep residual learning, improves the detail quality and perceptual realism of generated images, enabling ESRGAN to excel in image super-resolution tasks.

[0056] In the embodiment of the present invention, Figure 2 As shown, S1 includes:

[0057] Low-resolution images The input is sent to the generator, where the initial features are first extracted through the convolution layer. Then, deep features are extracted through multiple basic blocks to improve the network's expressive power. Skip connections are then used to effectively transfer shallow layer information to deep layers, improving training stability. Upsampling is performed at the back end of the multi-discriminator generative adversarial network to restore high-resolution features, and details are optimized through multiple convolution layers to ultimately output a super-resolution image. .

[0058] S2, super-resolution image and high-resolution real images The multiple discriminators are input together for discrimination, and the outputs of each discriminator are weighted and fused to obtain a comprehensive discrimination result.

[0059] In the embodiment of the present invention, Figure 3 and Figure 4 As shown, S2 includes:

[0060] The single discriminator of the generative adversarial network ESRGAN is improved into a multi-discriminator, which includes the basic discriminator Discriminator1, the feature clustering discriminator Discriminator2, and the feature block discriminator Discriminator3; the basic discriminator is the discriminator in ESRGAN, the feature clustering discriminator includes feature clustering, routing mechanism, and expert network; the feature block discriminator includes region adaptive processing, routing mechanism, and expert network;

[0061] In this embodiment of the present invention, the basic discriminator is used to determine global image quality and ensure overall visual consistency. The feature clustering discriminator combines MiniBatch K-Means clustering to classify input features and enhance feature differentiation. The feature block discriminator uses a sliding window strategy to partition local regions, improving edge detail recovery. Both the feature clustering discriminator and the feature block discriminator utilize expert networks and a feature routing mechanism for feature optimization. Each expert network focuses on different features (such as texture, edge, and color), enhancing multimodal processing capabilities. The feature clustering discriminator preprocesses input data through feature clustering (ISKM) to optimize expert network training, ensure feature matching, and avoid feature confounding. The feature block discriminator uses region-adaptive processing (ITS) to divide the image into small blocks and independently model local features. This improves detail recovery in complex structural areas and reduces artifacts and over-smoothing.

[0062] The basic discriminator process includes adopting a convolutional neural network (CNN)-based architecture, which consists of multiple convolutional layers and activation functions Leaky ReLU, gradually extracting image features, and using fully connected layers and activation functions Sigmoid to discriminate whether the input image is a super-resolution image. or high-resolution real images ;

[0063] The process of feature clustering discriminator and feature block discriminator includes feature extraction module FEM, spectral clustering module ISKM and image blocker ITS, which respectively perform super-resolution image classification on the input. and high-resolution real images After processing by the feature routing module FRM, the feature map is obtained. to , and through n expert sub-networks to After further processing, the comprehensive discrimination result is finally obtained through the feature selection module CFS.

[0064] In the embodiment of the present invention, the spectral clustering module ISKM includes:

[0065] ISKM uses a container-filling-based strategy to cluster features and dynamically adjusts the center value of the cluster to make the clustering result more stable and robust. Its expression is:

[0066] ;

[0067] Each cluster As a container, used to store the assigned pixel features ,in Indicates the number of pixels filled, is the center of the cluster, During the clustering process, new pixel features are assigned to the nearest cluster based on the distance from the cluster center, and the mean is updated. When the container is not full, new pixels are added directly and the mean is updated. When the container is full, existing pixels are replaced to keep the cluster size stable.

[0068] In the embodiment of the present invention, the process of the image blocker module ITS includes:

[0069] ITS extracts local features through sliding windows and reorganizes data using extended Unfold operations to improve feature expression capabilities and adapt to subsequent clustering analysis; input feature maps The shape is ,in is the batch size, is the number of channels, is the feature map size; ITS is through the window size and step length Extract local area, if The windows do not overlap if There is overlap, which helps to enhance the ability to capture local features; then, ITS uses the expanded Unfold operation to The region is flattened to -dimensional feature vector and convert the entire feature map into shape tensor, where is the total number of windows, calculated as This conversion method not only enhances the ability to express local features, but also optimizes the data format, making it more suitable for subsequent spectral clustering modules (ISKM, etc.), improving the robustness and adaptability of clustering. At the same time, it can flexibly adapt to different tasks such as super-resolution and object detection by adjusting P and S.

[0070] In the embodiment of the present invention, the process of the feature routing module FRM includes:

[0071] FRM uses the routing matrix to assign the optimal expert subnetwork ESN to the feature map processed by ISKM or ITS to improve the feature expression ability and computational efficiency. First, the input feature map Through the learnable weight matrix Calculate the routing matrix R, and obtain the distribution weight of each sub-network through Softmax normalization; then, according to the maximum matching degree , each feature map The most suitable expert sub-network is sent for processing. The expert sub-network is specially optimized for different feature types (such as texture, edge or structural information). The processed features are aggregated by the feature selection module CFS in the final fusion stage to form global features. Among them, the weight matrix Input into the routing mechanism to obtain the routing matrix R, which is expressed as: , NULL means invalid.

[0072] In the embodiment of the present invention, feature clustering includes:

[0073] The K-means++ initialization algorithm is used to select the initial cluster center to improve clustering convergence; the allocation of each feature vector prioritizes its distance from the cluster center and is adjusted according to feature similarity; the loss function is set to make the intra-cluster distance as small as possible and the inter-cluster distance as large as possible, thereby expanding the independence between different features.

[0074] In the embodiment of the present invention, the regional adaptive processing includes:

[0075] The input feature map is divided into multiple small blocks according to a predetermined size; each small block includes a local area in the feature map and is sent to the subsequent network module as an independent input for processing.

[0076] In the embodiment of the present invention, the expert network is composed of multiple sub-networks and a routing weight. Each sub-network processes a specific type of data, and the routing network is responsible for distributing input data to the expert sub-network according to the routing weight.

[0077] In an embodiment of the present invention, in order to constrain the expert subnetwork and the two data preprocessing methods, a feature density loss function and a load consistency loss function are set when training the network, and the optimal price of the problem is found by minimizing the loss value.

[0078] S3. Based on the comprehensive judgment results, use the loss function to calculate the loss and obtain the loss value.

[0079] In the embodiment of the present invention, the loss of the generator includes pixel loss , reconstruction losses , fight against losses etc. The loss of the discriminator includes feature density loss , Expert Load Consistency Loss wait.

[0080] In the embodiment of the present invention, S3 includes:

[0081] Adopt reconstruction loss and perceptual loss to optimize low-resolution images With super-resolution images Mapping between them to improve image quality; calculate super-resolution images With high-resolution real images The perceptual loss between is used to ensure the perceptual quality of the generated image; the intra-class feature compactness loss is calculated , used to reduce the intra-class feature dispersion and improve feature discrimination; calculate the inter-class feature dispersion loss , used to increase the spacing between features of different categories; calculate expert load consistency loss , used to balance the task distribution of expert networks and improve network stability; by integrating the previous adversarial loss , construct the discriminator loss function , whose expression is:

[0082] ;

[0083] in, 、 and is the weight coefficient; It is a feature density loss, which is used to optimize the feature distribution, making similar features more compact and heterogeneous features more dispersed, thereby improving the discriminative ability of the model. It includes intra-class feature compactness loss and inter-class feature dispersion loss , whose expression is: ; and is the weight coefficient, which is used to adjust the influence of the two parts of loss.

[0084] In this embodiment of the present invention, the intra-class feature compactness loss is used to reduce the dispersion of intra-class features, so that feature points of the same category are more concentrated in the feature space. Its expression is:

[0085] ;

[0086] in, and Represents the feature vector of similar samples. The loss calculation uses the Gaussian kernel function to measure the similarity of each pair of samples; Euclidean distance The smaller the value, the closer the exponential function is to 1, indicating that the samples within the class are closer together. Since this term takes a negative value, the model tends to minimize the loss during optimization, making the features within the class more compact. Intuitively, this loss term encourages features of samples in the same class to move closer to the center, forming a tighter intra-class distribution, reducing feature discreteness, and preventing the impact of feature dispersion on classification performance.

[0087] The inter-class feature dispersion loss is used to increase the feature spacing between different categories, so that the features of different categories are distributed far apart, thereby improving the separability of classification and avoiding the impact of feature overlap on classification performance. Its expression is:

[0088] ;

[0089] in, and Represents the normalized feature vectors of different categories. The inter-class feature dispersion loss calculates the cosine similarity of feature points between all categories and calculates the average value. Since we hope that the features of different categories are as different as possible, the optimization goal is to make The value of is as small as possible, that is, the angle between features of different categories is as large as possible. Intuitively, this loss term encourages features of different categories to be as far apart as possible in the feature space, reducing the overlap of features between classes and thus improving the discriminative ability of the model.

[0090] In the embodiment of the present invention, the expert load consistency loss (Load Equilibrium Loss, referred to as ) aims to optimize the load balancing of different expert subnetworks (ESNs) during the network allocation process, ensuring that all expert subnetworks can evenly share tasks during feature processing and avoiding excessive concentration or waste of computing resources. This loss measures the task allocation of different expert networks and adjusts the feature allocation strategy to balance the task load of all expert subnetworks, thereby improving the network's computational efficiency and stability. Its expression is:

[0091] ;

[0092] in, represents the number of expert sub-networks, represents the total number of input feature blocks, Reflect the The feature blocks are assigned to The probability of an expert sub-network, As a binary indicator, it is used to indicate whether the feature block is actually assigned to the expert sub-network; the calculation method of expert load consistency loss is to first calculate the average allocation weight of each expert and actual distribution , then taking their product and weighted normalization to measure overall load balance. The optimization goal of this loss is to minimize it so that each expert subnetwork receives a balanced task distribution, avoiding overloading some experts and underutilizing others. The introduction of load consistency loss helps improve network stability, enhance the computational efficiency of the model, and strengthen the synergy between different subnetworks.

[0093] S4. Update the parameters based on the loss value and gradient descent algorithm to optimize the multi-discriminator generative adversarial network.

[0094] In the embodiment of the present invention, the generator optimizes parameters according to the adversarial loss feedback from the discriminator to improve the super-resolution image. The discriminator is updated at the same time to improve the quality of super-resolution images With high-resolution real images discrimination ability for image reconstruction.

[0095] In the embodiment of the present invention, the updating or training process includes: Input to the generator to generate super-resolution images ; Super-resolution image High-resolution real images The multi-discriminator is input together for discrimination, and the output of each discriminator is weighted and fused to obtain a comprehensive discrimination result; based on the comprehensive discrimination result, the loss is calculated using the loss function, and the weights of the multi-discriminator generative adversarial network are updated through the gradient descent algorithm; the above steps are repeated until the number of training rounds of the network reaches the threshold or the loss value reaches the minimum.

[0096] S5. Repeat the above steps until the iteration is completed or the loss value reaches the minimum, and the optimal weight is obtained, and it is applied to the multi-discriminator generative adversarial network. At this time, the low-resolution image Input to the generator to obtain the final super-resolution image.

[0097] In the embodiment of the present invention, the final super-resolution image is output as the final result, achieving high-quality image magnification and detail restoration.

[0098] In this embodiment of the present invention, the DIV2K dataset (800 high-quality images) was used as the training set. Low-resolution (LR) images were generated through bicubic downsampling (×4). During training, the images were randomly cropped to 128×128 and data augmentation was performed. The test set included Set5, Set14, BSD100, and Urban100. The model was evaluated using PSNR and SSIM as pixel-level metrics, and LPIPS and DISTS to assess perceptual quality, comprehensively measuring the model's reconstruction performance and adaptability.

[0099] To prevent the generator from crashing due to the discriminator being too strong, the pre-training weights are used before formal training. The balance weights of LG and LD are set to (0.01, 1, 0.05) and (1, 10, 0.05) respectively. The hyperparameters Set to (1, 0.1, 0.1). The model uses Adam optimizer, and the learning rate is Initially, training was done for 60,000 iterations using a 4060 GPU, which took approximately 47 hours.

[0100] In this embodiment of the present invention, the performance of 12 mainstream models (SRCNN and FSRCNN, based on CNN as the baseline framework; SRGAN, USRGAN, BSRGAN, and ESRGAN, based on GAN as the baseline framework; and models based on SwinIR and RRBD) was compared on four datasets, as shown in Table 1. Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were used as evaluation metrics. The results in Table 1 show significant differences between the different methods on each dataset. Traditional super-resolution models SRCNN and FSRCNN, which use the MSE loss, achieve high scores in PSNR and SSIM, but their reconstructed images are overly smoothed and lack detail. In comparison, GAN methods such as SRGAN and ESRGAN score slightly lower, but achieve better visual perception. The SwirlIR and RRDB series of models, utilizing a multi-scale residual dense architecture and Transformer-CNN design, achieve excellent PSNR / SSIM performance while also ensuring good visual quality. The FCR-ESRGAN proposed in this paper outperforms other methods in terms of PSNR and SSIM on the Set5, Set14, Mangal109 and Urban100 datasets, thanks to the optimization of complex features by the multi-discriminant architecture and expert network.

[0101] Table 1 Comparison of PSNR and SSIM of different models

[0102] .

[0103] In the embodiment of the present invention, the existing methods (HR, SwinlR, BSRGAN, ESRGAN, USRGAN) are compared with FCR-ESRGAN in the classic image super-resolution task. Figure 5 As shown in the figure, images restored by other methods contain false textures, such as the distorted bricks in the first row, the false textures of the building walls in the second row, and the unnatural brick textures in the last row. Through these comparisons, it can be observed that images restored by other methods may contain unrealistic textures, while the method of the present invention can generate more natural and detailed results, eliminating artifacts and enhancing the realism of texture details. This proves that the multi-discrimination using expert networks has a stronger ability to distinguish detailed textures.

[0104] The present invention adopts multiple discriminators to replace the single discriminator of the traditional model, and adds an expert network, a routing mechanism and two data preprocessing mechanisms to the discriminator. The purpose is to solve the problem that the single discriminator of the traditional model cannot cope with the complexity and diversity of image features in the real world, and to improve the generalization and stability of the model.

[0105] Compared with the prior art, the present invention has the following beneficial effects:

[0106] (1) The multi-discriminator architecture of the present invention has stronger feature extraction and recognition capabilities, solving the problem that traditional single discriminators are difficult to take into account both global and local features, and improving the detail restoration quality and authenticity of super-resolution images.

[0107] (2) The feature clustering discriminator (Discriminator2) of the present invention has more accurate feature classification and processing capabilities, which solves the problem of reduced generation quality caused by the mixing of features of different types of images, and improves the generalization ability of the model and its adaptability to complex textures.

[0108] (3) The feature block discriminator (Discriminator3) of the present invention has a more accurate local area feature modeling capability, solves the problems of blurred edge details and loss of high-frequency information, and improves the detail restoration capability of image reconstruction.

[0109] (4) The expert network and routing mechanism of the present invention have the ability to adaptively optimize for different feature categories, solving the problem of single optimization direction of traditional discriminators when processing multimodal features, and improving the quality and diversity of image super-resolution reconstruction.

[0110] (5) The feature clustering (ISKM) preprocessing of the present invention has the effect of enhancing the independence of input data and optimizing discriminant learning, solving the problem of unstable training caused by inaccurate feature classification, and improving the stability of adversarial training and the consistency of image generation.

[0111] (6) The regional adaptive processing (ITS) of the present invention has a more accurate local area feature recovery capability, solves the problems of local detail loss and insufficient texture reconstruction, and improves the restoration quality of complex image areas (such as edges and high-frequency texture areas).

[0112] In the technical solution provided by the present invention, the method includes inputting a low-resolution image into a generator to generate a super-resolution image; inputting the super-resolution image and the high-resolution real image into a multi-discriminator for discrimination, and the outputs of each discriminator are weighted and fused respectively through weights to obtain a comprehensive discrimination result; according to the comprehensive discrimination result, the loss is calculated using a loss function to obtain a loss value; according to the loss value and the gradient descent algorithm, the parameters are updated to optimize the multi-discriminator generative adversarial network; repeating the above steps until the iteration is completed or the loss value reaches the minimum, the optimal weight is obtained, and it is applied to the multi-discriminator generative adversarial network, at this time the low-resolution image is input into the generator to obtain the final super-resolution image. This method solves the limitation problem of the single discriminator in the traditional model and improves the generalization and stability of the network.

[0113] Each step of the embodiment of the present invention may be performed by an electronic device, including but not limited to a mobile phone, a tablet computer, a portable PC, a desktop computer, etc.

[0114] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the electronic device where the computer-readable storage medium is located is controlled to execute the above-mentioned embodiment of the feature clustering and region-adaptive multi-discriminator generative adversarial network enhancement method.

[0115] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the electronic device 21 includes: a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, the feature clustering and region-adaptive multi-discriminator generative adversarial network enhancement method in the embodiment is implemented. To avoid repetition, they are not described here one by one.

[0116] The electronic device 21 includes, but is not limited to, a processor 211 and a memory 212. Those skilled in the art will understand that Figure 6 It is only an example of the electronic device 21 and does not constitute a limitation of the electronic device 21. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0117] The processor 211 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0118] The memory 212 can be an internal storage unit of the electronic device 21, such as the hard drive or memory of the electronic device 21. The memory 212 can also be an external storage device of the electronic device 21, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 21. Furthermore, the memory 212 can include both the internal storage unit of the electronic device 21 and an external storage device. The memory 212 is used to store computer programs and other programs and data required by the network device. The memory 212 can also be used to temporarily store data that has been output or is about to be output.

[0119] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A feature clustering and region-adaptive multi-discriminator generative adversarial network enhancement method, characterized by: The method comprises: S1, low resolution image Input to the generator to generate super-resolution images ; S2, super-resolution image and high-resolution real images The multiple discriminators are input together for discrimination, and the outputs of each discriminator are weighted and fused to obtain a comprehensive discrimination result; S3. Based on the comprehensive discrimination results, use the loss function to calculate the loss and obtain the loss value; S4. Update parameters based on the loss value and gradient descent algorithm to optimize the multi-discriminator generative adversarial network; S5. Repeat the above steps until the iteration is completed or the loss value reaches the minimum, and the optimal weight is obtained, and it is applied to the multi-discriminator generative adversarial network. At this time, the low-resolution image Input to the generator to obtain the final super-resolution image; The S2 includes: The single discriminator of the generative adversarial network ESRGAN is improved into a multi-discriminator, which includes the basic discriminator Discriminator1, the feature clustering discriminator Discriminator2, and the feature block discriminator Discriminator3; the basic discriminator is the discriminator in ESRGAN, the feature clustering discriminator includes feature clustering, routing mechanism, and expert network; the feature block discriminator includes region adaptive processing, routing mechanism, and expert network; The basic discriminator process includes adopting a convolutional neural network (CNN)-based architecture, which consists of multiple convolutional layers and activation functions Leaky ReLU, gradually extracting image features, and using fully connected layers and activation functions Sigmoid to discriminate whether the input image is a super-resolution image. or high-resolution real images ; The process of feature clustering discriminator and feature block discriminator includes feature extraction module FEM, spectral clustering module ISKM and image blocker ITS, which respectively perform super-resolution image classification on the input. and high-resolution real images After processing by the feature routing module FRM, the feature map is obtained. to , and through n expert sub-networks to After further processing, the comprehensive discrimination result is finally obtained through the feature selection module CFS.

2. The method according to claim 1, characterized in that Said S1 comprises: Low-resolution images The input is fed into the generator, where the initial features are first extracted through the convolutional layer. Then, deep features are extracted through multiple basic blocks. Skip connections are then used to effectively transfer shallow layer information to deep layers. Upsampling is performed at the back end of the multi-discriminator generative adversarial network to restore high-resolution features, and details are optimized through multiple convolutional layers to ultimately output a super-resolution image. .

3. The method according to claim 1, characterized in that The spectral clustering module ISKM includes: ISKM uses a container-filling-based strategy to cluster features and dynamically adjusts the center value of the cluster to make the clustering result more stable and robust. Its expression is: ； Each cluster As a container, used to store the assigned pixel features ,in Indicates the number of pixels filled, is the center of the cluster, During the clustering process, new pixel features are assigned to the nearest cluster based on the distance from the cluster center, and the mean is updated. When the container is not full, new pixels are added directly and the mean is updated. When the container is full, existing pixels are replaced to keep the cluster size stable.

4. The method according to claim 1, wherein The process of the image segmenter ITS includes: ITS extracts local features through a sliding window and reorganizes the data using an extended Unfold operation to improve feature expression capabilities and adapt to subsequent clustering analysis; input feature map The shape is ,in is the batch size, is the number of channels, is the feature map size; ITS is through the window size and step length Extract local area, if The windows do not overlap if There is overlap, which helps to enhance the ability to capture local features; then, ITS uses the expanded Unfold operation to The region is flattened to -dimensional feature vector and convert the entire feature map into shape tensor, where is the total number of windows, calculated as .

5. The method according to claim 1, wherein The process of the feature routing module FRM includes: FRM uses the routing matrix to assign the optimal expert subnetwork ESN to the feature map processed by ISKM or ITS to improve the feature expression ability and computational efficiency. First, the input feature map Through the learnable weight matrix Calculate the routing matrix R, and obtain the distribution weight of each sub-network through Softmax normalization; then, according to the maximum matching degree , each feature map The features are sent to the most suitable expert sub-network for processing; the processed features are aggregated in the final fusion stage through the feature selection module CFS to form global features; among them, the weight matrix Input into the routing mechanism to obtain the routing matrix R, which is expressed as: , NULL means invalid.

6. The method according to claim 1, characterized in that The S3 includes: Adopt reconstruction loss and perceptual loss to optimize low-resolution images With super-resolution images Mapping between them to improve image quality; calculate super-resolution images With high-resolution real images The perceptual loss between is used to ensure the perceptual quality of the generated image; the intra-class feature compactness loss is calculated , used to reduce the intra-class feature dispersion; calculate the inter-class feature dispersion loss , used to increase the spacing between features of different categories; calculate expert load consistency loss , used to balance the task allocation of expert networks; by integrating the previous adversarial loss , construct the discriminator loss function , whose expression is: ； in, 、 and is the weight coefficient; is the feature density loss, including the intra-class feature compactness loss and inter-class feature dispersion loss , whose expression is: ; and is the weight coefficient, which is used to adjust the influence of the two parts of loss.

7. The method according to claim 6, characterized in that The intra-class feature compactness loss is used to reduce the dispersion of intra-class features so that feature points of the same category are more concentrated in the feature space. Its expression is: ； in, and Represents the feature vector of similar samples. The loss calculation uses the Gaussian kernel function to measure the similarity of each pair of samples; Euclidean distance The smaller it is, the closer the value of the exponential function is to 1, indicating that the samples within the class are closer; The inter-class feature dispersion loss is used to increase the feature spacing of different categories, so that the features of different categories are distributed far away, thereby improving the separability of classification. Its expression is: ； in, and Represents the normalized feature vectors of different categories. The inter-class feature dispersion loss calculates the cosine similarity of feature points between all categories and takes the average value.

8. The method according to claim 6, characterized in that The expert load consistency loss It aims to optimize the load balancing of different expert sub-networks ESN during network allocation and ensure that all expert sub-networks can evenly share the tasks during feature processing. Its expression is: ； in, represents the number of expert sub-networks, represents the total number of input feature blocks, Reflect the The feature blocks are assigned to The probability of an expert sub-network, As a binary indicator, it is used to indicate whether the feature block is actually assigned to the expert sub-network; the calculation method of expert load consistency loss is to first calculate the average allocation weight of each expert and actual distribution , then take their product and weight normalize them to measure the overall load balancing.

Images